2023-11-21 17:02:47,741 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-21 17:02:48,162 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-21 17:02:48,347 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-21 17:02:48,438 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-21 17:02:48,446 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-21 17:02:48,463 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=639, ProcessCount=167, AvailableMemoryMB=7064
2023-11-21 17:02:48,466 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-21 17:02:48,466 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-21 17:02:48,647 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101776415:84 , srv1020947857:84 , srv1079027675:84 , srv1091522098:84 , srv1122749591:84 , srv1137142874:84 , srv1138839217:84 , srv1167405475:84 , srv1177084601:84 , srv1204915642:84 , srv1223181026:84 , srv1225593425:84 , srv123001013:84 , srv1232447958:84 , srv1291005463:84 , srv1302008874:84 , srv1315032143:84 , srv1340216326:84 , srv1342524126:84 , srv1369599019:84 , srv1394375565:84 , srv1404644909:84 , srv1406705734:84 , srv1409093035:84 , srv140919180:84 , srv1440185004:84 , srv1447051999:84 , srv1447392286:84 , srv1477555204:84 , srv1484766444:84 , srv1583622882:84 , srv1584598494:84 , srv1599246014:84 , srv1600378281:84 , srv1601729019:84 , srv1601819663:84 , srv1604303487:84 , srv1632807692:84 , srv1636917910:84 , srv165017599:84 , srv166904965:84 , srv1686049806:84 , srv1688046808:84 , srv1689166362:84 , srv1711333122:84 , srv171685076:84 , srv1720831510:84 , srv1727187108:84 , srv1735637933:84 , srv175426936:84 , srv1761196331:84 , srv1769172313:84 , srv1789951166:84 , srv1819939201:84 , srv1827465579:84 , srv1847090469:84 , srv1899382391:84 , srv1900460836:84 , srv1903621326:84 , srv1939058585:84 , srv1948884955:84 , srv1958157770:84 , srv1966915598:84 , srv1970445482:84 , srv2049073990:84 , srv207163059:84 , srv2075933200:84 , srv2096312348:84 , srv211289442:84 , srv219915955:84 , srv265509165:84 , srv301160101:84 , srv311844507:84 , srv367987624:84 , srv370373565:84 , srv437912625:84 , srv464336562:84 , srv468957009:84 , srv4864046:84 , srv494786636:84 , srv519560432:84 , srv532093445:84 , srv535539703:84 , srv546187120:84 , srv572177968:84 , srv577312003:84 , srv646468449:84 , srv712387546:84 , srv725928437:84 , srv735410450:84 , srv763524426:84 , srv782761002:84 , srv906482407:84 , srv930561149:84 , srv946314210:84 , srv965651555:84 , srv98771912:84 , srv989811155:84 , srv990037205:84 , srv22462622:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-21 17:02:48,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-21 17:02:48,669 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1404644909=21, srv1903621326=58, srv989811155=98, srv123001013=12, srv1406705734=22, srv782761002=92, srv1079027675=2, srv1447051999=26, srv763524426=91, srv2049073990=64, srv494786636=80, srv1583622882=30, srv140919180=24, srv646468449=87, srv1177084601=8, srv1601819663=35, srv1484766444=29, srv1966915598=62, srv1970445482=63, srv712387546=88, srv1939058585=59, srv101776415=0, srv946314210=95, srv1636917910=38, srv990037205=99, srv2075933200=66, srv1340216326=17, srv1948884955=60, srv1138839217=6, srv519560432=81, srv1847090469=55, srv1827465579=54, srv577312003=86, srv1599246014=32, srv311844507=73, srv1302008874=15, srv1223181026=10, srv1020947857=1, srv1761196331=50, srv1440185004=25, srv1600378281=33, srv1315032143=16, srv1409093035=23, srv1584598494=31, srv175426936=49, srv1122749591=4, srv1720831510=46, srv1167405475=7, srv1137142874=5, srv1689166362=43, srv207163059=65, srv1789951166=52, srv1369599019=19, srv1225593425=11, srv1899382391=56, srv1735637933=48, srv22462622=70, srv1204915642=9, srv1727187108=47, srv1819939201=53, srv1769172313=51, srv98771912=97, srv1447392286=27, srv1900460836=57, srv367987624=74, srv468957009=78, srv1477555204=28, srv906482407=93, srv546187120=84, srv735410450=90, srv1291005463=14, srv2096312348=67, srv1232447958=13, srv1601729019=34, srv965651555=96, srv1091522098=3, srv1632807692=37, srv171685076=45, srv535539703=83, srv265509165=71, srv572177968=85, srv1711333122=44, srv930561149=94, srv532093445=82, srv1686049806=41, srv211289442=68, srv301160101=72, srv1958157770=61, srv219915955=69, srv1394375565=20, srv166904965=40, srv1342524126=18, srv437912625=76, srv370373565=75, srv725928437=89, srv165017599=39, srv464336562=77, srv1688046808=42, srv1604303487=36, srv4864046=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-21 17:02:48,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-21 17:02:48,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-21 17:02:48,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-21 17:02:48,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-21 17:02:48,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-21 17:02:48,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-21 17:02:48,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-21 17:02:48,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-21 17:02:48,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-21 17:02:48,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-21 17:02:48,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-21 17:02:48,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-21 17:02:48,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-21 17:02:48,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-21 17:02:48,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-21 17:02:48,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-21 17:02:48,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-21 17:02:48,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-21 17:02:48,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-21 17:02:48,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-21 17:02:48,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-21 17:02:48,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-21 17:02:48,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-21 17:02:48,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-21 17:02:48,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-21 17:02:48,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-21 17:02:48,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-21 17:02:48,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-21 17:02:48,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-21 17:02:48,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-21 17:02:48,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-21 17:02:48,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-21 17:02:48,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-21 17:02:48,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-21 17:02:48,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-21 17:02:48,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-21 17:02:48,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-21 17:02:48,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-21 17:02:48,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-21 17:02:48,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-21 17:02:48,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-21 17:02:48,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-21 17:02:48,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-21 17:02:48,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-21 17:02:48,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-21 17:02:48,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-21 17:02:48,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-21 17:02:48,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-21 17:02:48,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-21 17:02:48,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-21 17:02:48,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-21 17:02:48,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-21 17:02:48,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-21 17:02:48,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-21 17:02:48,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-21 17:02:48,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-21 17:02:48,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-21 17:02:48,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-21 17:02:48,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-21 17:02:48,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-21 17:02:48,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-21 17:02:48,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-21 17:02:48,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-21 17:02:48,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-21 17:02:48,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-21 17:02:48,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-21 17:02:48,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-21 17:02:48,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-21 17:02:48,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-21 17:02:48,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-21 17:02:48,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-21 17:02:48,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-21 17:02:48,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-21 17:02:48,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-21 17:02:48,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-21 17:02:48,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-21 17:02:48,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-21 17:02:48,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-21 17:02:48,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-21 17:02:48,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-21 17:02:48,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-21 17:02:48,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-21 17:02:48,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-21 17:02:48,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-21 17:02:48,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-21 17:02:48,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-21 17:02:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-21 17:02:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-21 17:02:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-21 17:02:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-21 17:02:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-21 17:02:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-21 17:02:48,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-21 17:02:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-21 17:02:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-21 17:02:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-21 17:02:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-21 17:02:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-21 17:02:48,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-21 17:02:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-21 17:02:48,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-21 17:02:49,066 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:49,066 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-21 17:02:49,067 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-21 17:02:49,440 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 596 ms to try 5000 different iterations.  Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17830942511408007. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04040911864831881, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05827293650631021, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04040911864831881, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17341666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:49,476 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1688046808:81 , srv1970445482:81 , srv1020947857:82 , srv1225593425:82 , srv1232447958:82 , srv1404644909:82 , srv1686049806:82 , srv1899382391:82 , srv906482407:82 , srv1447051999:83 , srv1447392286:83 , srv1761196331:83 , srv1769172313:83 , srv219915955:83 , srv532093445:83 , srv930561149:83 , srv1177084601:84 , srv1477555204:84 , srv1632807692:84 , srv1727187108:84 , srv1819939201:84 , srv1847090469:84 , srv2096312348:84 , srv265509165:84 , srv370373565:84 , srv646468449:84 , srv725928437:84 , srv1302008874:85 , srv140919180:85 , srv1600378281:85 , srv1636917910:85 , srv165017599:85 , srv171685076:85 , srv1966915598:85 , srv207163059:85 , srv4864046:85 , srv494786636:85 , srv577312003:85 , srv763524426:85 , srv782761002:85 , srv965651555:85 , srv98771912:85 , srv989811155:85 , srv1204915642:86 , srv1406705734:86 , srv1584598494:86 , srv1711333122:86 , srv1720831510:86 , srv1789951166:86 , srv1900460836:86 , srv1939058585:86 , srv2075933200:86 , srv1137142874:87 , srv1291005463:87 , srv1342524126:87 , srv1604303487:87 , srv175426936:87 , srv211289442:87 , srv311844507:87 , srv464336562:87 , srv546187120:87 , srv1138839217:88 , srv1315032143:88 , srv1340216326:88 , srv1369599019:88 , srv1394375565:88 , srv1409093035:88 , srv1583622882:88 , srv166904965:88 , srv1689166362:88 , srv1827465579:88 , srv367987624:88 , srv468957009:88 , srv101776415:89 , srv1167405475:89 , srv1223181026:89 , srv123001013:89 , srv1484766444:89 , srv1599246014:89 , srv1601819663:89 , srv1735637933:89 , srv1903621326:89 , srv1958157770:89 , srv2049073990:89 , srv437912625:89 , srv519560432:89 , srv535539703:89 , srv572177968:89 , srv1440185004:90 , srv1601729019:90 , srv712387546:90 , srv990037205:90 , srv1079027675:91 , srv1122749591:91 , srv735410450:91 , srv1948884955:92 , srv301160101:92 , srv946314210:92 , srv1091522098:93 , srv22462622:449 }
2023-11-21 17:02:49,482 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-21 17:02:49,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1404644909=21, srv1903621326=58, srv989811155=98, srv123001013=12, srv1406705734=22, srv782761002=92, srv1079027675=2, srv1447051999=26, srv763524426=91, srv2049073990=64, srv494786636=80, srv1583622882=30, srv140919180=24, srv646468449=87, srv1177084601=8, srv1601819663=35, srv1484766444=29, srv1966915598=62, srv1970445482=63, srv712387546=88, srv1939058585=59, srv101776415=0, srv946314210=95, srv1636917910=38, srv990037205=99, srv2075933200=66, srv1340216326=17, srv1948884955=60, srv1138839217=6, srv519560432=81, srv1847090469=55, srv1827465579=54, srv577312003=86, srv1599246014=32, srv311844507=73, srv1302008874=15, srv1223181026=10, srv1020947857=1, srv1761196331=50, srv1440185004=25, srv1600378281=33, srv1315032143=16, srv1409093035=23, srv1584598494=31, srv175426936=49, srv1122749591=4, srv1720831510=46, srv1167405475=7, srv1137142874=5, srv1689166362=43, srv207163059=65, srv1789951166=52, srv1369599019=19, srv1225593425=11, srv1899382391=56, srv1735637933=48, srv22462622=70, srv1204915642=9, srv1727187108=47, srv1819939201=53, srv1769172313=51, srv98771912=97, srv1447392286=27, srv1900460836=57, srv367987624=74, srv468957009=78, srv1477555204=28, srv906482407=93, srv546187120=84, srv735410450=90, srv1291005463=14, srv2096312348=67, srv1232447958=13, srv1601729019=34, srv965651555=96, srv1091522098=3, srv1632807692=37, srv171685076=45, srv535539703=83, srv265509165=71, srv572177968=85, srv1711333122=44, srv930561149=94, srv532093445=82, srv1686049806=41, srv211289442=68, srv301160101=72, srv1958157770=61, srv219915955=69, srv1394375565=20, srv166904965=40, srv1342524126=18, srv437912625=76, srv370373565=75, srv725928437=89, srv165017599=39, srv464336562=77, srv1688046808=42, srv1604303487=36, srv4864046=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-21 17:02:49,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-21 17:02:49,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-21 17:02:49,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-21 17:02:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-21 17:02:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-21 17:02:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-21 17:02:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-21 17:02:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-21 17:02:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-21 17:02:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-21 17:02:49,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-21 17:02:49,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-21 17:02:49,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-21 17:02:49,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-21 17:02:49,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-21 17:02:49,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-21 17:02:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-21 17:02:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-21 17:02:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-21 17:02:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-21 17:02:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-21 17:02:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-21 17:02:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-21 17:02:49,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-21 17:02:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-21 17:02:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-21 17:02:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-21 17:02:49,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-21 17:02:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-21 17:02:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-21 17:02:49,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-21 17:02:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-21 17:02:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-21 17:02:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-21 17:02:49,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-21 17:02:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-21 17:02:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-21 17:02:49,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-21 17:02:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-21 17:02:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-21 17:02:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-21 17:02:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-21 17:02:49,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-21 17:02:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-21 17:02:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-21 17:02:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-21 17:02:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-21 17:02:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-21 17:02:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-21 17:02:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-21 17:02:49,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-21 17:02:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-21 17:02:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-21 17:02:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-21 17:02:49,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-21 17:02:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-21 17:02:49,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-21 17:02:49,808 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04040911864831881, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05827293650631021, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04040911864831881, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17341666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:49,808 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-21 17:02:49,809 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17827652901497767, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04040911864831881, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05827293650631021, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04040911864831881, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17341666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.241, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-21 17:02:50,198 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 623 ms to try 5000 different iterations.  Found a solution that moves 2626 regions; Going from a computed imbalance of 0.17827652901497767 to a new imbalance of 0.014960276069289414. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021265605172133578); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029979416807182312, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2917777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021265605172133578); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:50,231 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1079027675:80 , srv1440185004:80 , srv1601729019:80 , srv535539703:80 , srv946314210:80 , srv1091522098:81 , srv123001013:81 , srv1948884955:81 , srv301160101:81 , srv735410450:81 , srv1122749591:82 , srv1340216326:82 , srv1369599019:82 , srv1484766444:82 , srv1735637933:82 , srv437912625:82 , srv572177968:82 , srv990037205:82 , srv101776415:83 , srv1138839217:83 , srv1223181026:83 , srv1394375565:83 , srv166904965:83 , srv1689166362:83 , srv1903621326:83 , srv2049073990:83 , srv468957009:83 , srv712387546:83 , srv1167405475:84 , srv1315032143:84 , srv1583622882:84 , srv1601819663:84 , srv1900460836:84 , srv1958157770:84 , srv519560432:84 , srv546187120:84 , srv1291005463:85 , srv1342524126:85 , srv1409093035:85 , srv1599246014:85 , srv1720831510:85 , srv175426936:85 , srv1827465579:85 , srv1966915598:85 , srv2075933200:85 , srv211289442:85 , srv367987624:85 , srv494786636:85 , srv1137142874:86 , srv1204915642:86 , srv1302008874:86 , srv1477555204:86 , srv1711333122:86 , srv171685076:86 , srv1847090469:86 , srv1406705734:87 , srv140919180:87 , srv1584598494:87 , srv1604303487:87 , srv1636917910:87 , srv1789951166:87 , srv207163059:87 , srv464336562:87 , srv930561149:87 , srv989811155:87 , srv1447392286:88 , srv1600378281:88 , srv165017599:88 , srv1939058585:88 , srv2096312348:88 , srv311844507:88 , srv4864046:88 , srv532093445:88 , srv577312003:88 , srv725928437:88 , srv763524426:88 , srv906482407:88 , srv98771912:88 , srv1177084601:89 , srv1232447958:89 , srv1447051999:89 , srv1632807692:89 , srv1686049806:89 , srv1769172313:89 , srv1970445482:89 , srv219915955:89 , srv782761002:89 , srv965651555:89 , srv1020947857:90 , srv1225593425:90 , srv1404644909:90 , srv1727187108:90 , srv1761196331:90 , srv370373565:90 , srv646468449:90 , srv265509165:91 , srv1688046808:92 , srv1819939201:92 , srv1899382391:92 , srv22462622:514 }
2023-11-21 17:02:50,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-21 17:02:50,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1404644909=21, srv1903621326=58, srv989811155=98, srv123001013=12, srv1406705734=22, srv782761002=92, srv1079027675=2, srv1447051999=26, srv763524426=91, srv2049073990=64, srv494786636=80, srv1583622882=30, srv140919180=24, srv646468449=87, srv1177084601=8, srv1601819663=35, srv1484766444=29, srv1966915598=62, srv1970445482=63, srv712387546=88, srv1939058585=59, srv101776415=0, srv946314210=95, srv1636917910=38, srv990037205=99, srv2075933200=66, srv1340216326=17, srv1948884955=60, srv1138839217=6, srv519560432=81, srv1847090469=55, srv1827465579=54, srv577312003=86, srv1599246014=32, srv311844507=73, srv1302008874=15, srv1223181026=10, srv1020947857=1, srv1761196331=50, srv1440185004=25, srv1600378281=33, srv1315032143=16, srv1409093035=23, srv1584598494=31, srv175426936=49, srv1122749591=4, srv1720831510=46, srv1167405475=7, srv1137142874=5, srv1689166362=43, srv207163059=65, srv1789951166=52, srv1369599019=19, srv1225593425=11, srv1899382391=56, srv1735637933=48, srv22462622=70, srv1204915642=9, srv1727187108=47, srv1819939201=53, srv1769172313=51, srv98771912=97, srv1447392286=27, srv1900460836=57, srv367987624=74, srv468957009=78, srv1477555204=28, srv906482407=93, srv546187120=84, srv735410450=90, srv1291005463=14, srv2096312348=67, srv1232447958=13, srv1601729019=34, srv965651555=96, srv1091522098=3, srv1632807692=37, srv171685076=45, srv535539703=83, srv265509165=71, srv572177968=85, srv1711333122=44, srv930561149=94, srv532093445=82, srv1686049806=41, srv211289442=68, srv301160101=72, srv1958157770=61, srv219915955=69, srv1394375565=20, srv166904965=40, srv1342524126=18, srv437912625=76, srv370373565=75, srv725928437=89, srv165017599=39, srv464336562=77, srv1688046808=42, srv1604303487=36, srv4864046=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-21 17:02:50,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-21 17:02:50,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-21 17:02:50,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-21 17:02:50,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-21 17:02:50,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-21 17:02:50,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-21 17:02:50,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-21 17:02:50,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-21 17:02:50,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-21 17:02:50,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-21 17:02:50,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-21 17:02:50,282 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-21 17:02:50,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-21 17:02:50,284 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-21 17:02:50,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-21 17:02:50,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-21 17:02:50,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-21 17:02:50,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-21 17:02:50,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-21 17:02:50,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-21 17:02:50,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-21 17:02:50,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-21 17:02:50,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-21 17:02:50,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-21 17:02:50,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-21 17:02:50,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-21 17:02:50,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-21 17:02:50,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-21 17:02:50,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-21 17:02:50,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-21 17:02:50,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-21 17:02:50,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-21 17:02:50,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-21 17:02:50,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-21 17:02:50,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-21 17:02:50,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-21 17:02:50,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-21 17:02:50,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-21 17:02:50,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-21 17:02:50,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-21 17:02:50,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-21 17:02:50,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-21 17:02:50,442 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021265605172133578); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029979416807182312, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021265605172133578); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:50,443 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-21 17:02:50,443 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014941888424282398, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021265605172133578); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.029979416807182312, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021265605172133578); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04583333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-21 17:02:50,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 515 ms to try 5000 different iterations.  Found a solution that moves 1050 regions; Going from a computed imbalance of 0.014941888424282398 to a new imbalance of 6.364656301278387E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005774582557741648); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.11666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005774582557741648); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:50,831 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1819939201:82 , srv265509165:82 , srv311844507:82 , srv1604303487:83 , srv370373565:83 , srv965651555:83 , srv1091522098:84 , srv1122749591:84 , srv1406705734:84 , srv1599246014:84 , srv1600378281:84 , srv1601819663:84 , srv165017599:84 , srv1727187108:84 , srv1789951166:84 , srv1899382391:84 , srv1958157770:84 , srv207163059:84 , srv301160101:84 , srv4864046:84 , srv519560432:84 , srv577312003:84 , srv646468449:84 , srv725928437:84 , srv782761002:84 , srv1020947857:85 , srv1137142874:85 , srv1138839217:85 , srv1167405475:85 , srv1177084601:85 , srv1223181026:85 , srv1225593425:85 , srv1232447958:85 , srv1315032143:85 , srv1342524126:85 , srv1404644909:85 , srv1409093035:85 , srv1584598494:85 , srv1632807692:85 , srv1636917910:85 , srv1686049806:85 , srv1688046808:85 , srv171685076:85 , srv175426936:85 , srv1761196331:85 , srv1827465579:85 , srv1939058585:85 , srv1948884955:85 , srv219915955:85 , srv367987624:85 , srv712387546:85 , srv763524426:85 , srv98771912:85 , srv989811155:85 , srv101776415:86 , srv1079027675:86 , srv1204915642:86 , srv1291005463:86 , srv1447051999:86 , srv1583622882:86 , srv1711333122:86 , srv1720831510:86 , srv1735637933:86 , srv1769172313:86 , srv1903621326:86 , srv2049073990:86 , srv2096312348:86 , srv211289442:86 , srv437912625:86 , srv464336562:86 , srv468957009:86 , srv532093445:86 , srv735410450:86 , srv946314210:86 , srv990037205:86 , srv123001013:87 , srv1302008874:87 , srv1340216326:87 , srv1394375565:87 , srv140919180:87 , srv1440185004:87 , srv1447392286:87 , srv1477555204:87 , srv1484766444:87 , srv1601729019:87 , srv166904965:87 , srv1689166362:87 , srv1900460836:87 , srv1966915598:87 , srv1970445482:87 , srv2075933200:87 , srv494786636:87 , srv546187120:87 , srv572177968:87 , srv930561149:87 , srv1369599019:88 , srv1847090469:88 , srv535539703:88 , srv906482407:88 , srv22462622:546 }
2023-11-21 17:02:50,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-21 17:02:50,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1404644909=21, srv1903621326=58, srv989811155=98, srv123001013=12, srv1406705734=22, srv782761002=92, srv1079027675=2, srv1447051999=26, srv763524426=91, srv2049073990=64, srv494786636=80, srv1583622882=30, srv140919180=24, srv646468449=87, srv1177084601=8, srv1601819663=35, srv1484766444=29, srv1966915598=62, srv1970445482=63, srv712387546=88, srv1939058585=59, srv101776415=0, srv946314210=95, srv1636917910=38, srv990037205=99, srv2075933200=66, srv1340216326=17, srv1948884955=60, srv1138839217=6, srv519560432=81, srv1847090469=55, srv1827465579=54, srv577312003=86, srv1599246014=32, srv311844507=73, srv1302008874=15, srv1223181026=10, srv1020947857=1, srv1761196331=50, srv1440185004=25, srv1600378281=33, srv1315032143=16, srv1409093035=23, srv1584598494=31, srv175426936=49, srv1122749591=4, srv1720831510=46, srv1167405475=7, srv1137142874=5, srv1689166362=43, srv207163059=65, srv1789951166=52, srv1369599019=19, srv1225593425=11, srv1899382391=56, srv1735637933=48, srv22462622=70, srv1204915642=9, srv1727187108=47, srv1819939201=53, srv1769172313=51, srv98771912=97, srv1447392286=27, srv1900460836=57, srv367987624=74, srv468957009=78, srv1477555204=28, srv906482407=93, srv546187120=84, srv735410450=90, srv1291005463=14, srv2096312348=67, srv1232447958=13, srv1601729019=34, srv965651555=96, srv1091522098=3, srv1632807692=37, srv171685076=45, srv535539703=83, srv265509165=71, srv572177968=85, srv1711333122=44, srv930561149=94, srv532093445=82, srv1686049806=41, srv211289442=68, srv301160101=72, srv1958157770=61, srv219915955=69, srv1394375565=20, srv166904965=40, srv1342524126=18, srv437912625=76, srv370373565=75, srv725928437=89, srv165017599=39, srv464336562=77, srv1688046808=42, srv1604303487=36, srv4864046=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-21 17:02:50,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-21 17:02:50,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-21 17:02:50,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-21 17:02:50,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-21 17:02:50,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-21 17:02:50,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-21 17:02:50,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-21 17:02:50,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-21 17:02:50,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-21 17:02:50,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-21 17:02:50,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-21 17:02:50,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-21 17:02:50,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-21 17:02:50,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-21 17:02:50,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-21 17:02:50,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-21 17:02:50,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-21 17:02:50,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-21 17:02:50,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-21 17:02:50,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-21 17:02:50,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-21 17:02:50,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-21 17:02:50,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-21 17:02:50,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-21 17:02:50,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-21 17:02:50,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-21 17:02:50,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-21 17:02:50,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-21 17:02:50,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-21 17:02:50,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-21 17:02:50,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-21 17:02:50,899 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-21 17:02:50,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-21 17:02:50,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-21 17:02:50,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-21 17:02:50,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-21 17:02:50,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-21 17:02:50,900 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-21 17:02:50,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-21 17:02:50,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-21 17:02:50,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-21 17:02:50,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-21 17:02:50,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-21 17:02:50,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-21 17:02:50,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-21 17:02:50,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-21 17:02:51,064 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005774582557741648); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005774582557741648); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:51,064 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-21 17:02:51,066 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.291133729848957E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005774582557741648); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006914582258636295); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005774582557741648); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004666666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-21 17:02:51,464 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 502 ms to try 5000 different iterations.  Found a solution that moves 259 regions; Going from a computed imbalance of 6.291133729848957E-4 to a new imbalance of 1.985899669972793E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.028777777777777777, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:51,467 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv464336562:82 , srv140919180:83 , srv1447051999:83 , srv2075933200:83 , srv101776415:84 , srv1091522098:84 , srv1137142874:84 , srv1167405475:84 , srv1177084601:84 , srv1204915642:84 , srv1291005463:84 , srv1302008874:84 , srv1342524126:84 , srv1369599019:84 , srv1394375565:84 , srv1409093035:84 , srv1440185004:84 , srv1447392286:84 , srv1484766444:84 , srv1584598494:84 , srv1599246014:84 , srv1632807692:84 , srv1636917910:84 , srv166904965:84 , srv1688046808:84 , srv1689166362:84 , srv1727187108:84 , srv1761196331:84 , srv1769172313:84 , srv1819939201:84 , srv1827465579:84 , srv1847090469:84 , srv1899382391:84 , srv1903621326:84 , srv1939058585:84 , srv1948884955:84 , srv1966915598:84 , srv2049073990:84 , srv211289442:84 , srv367987624:84 , srv572177968:84 , srv646468449:84 , srv712387546:84 , srv735410450:84 , srv763524426:84 , srv782761002:84 , srv906482407:84 , srv930561149:84 , srv946314210:84 , srv98771912:84 , srv990037205:84 , srv1020947857:85 , srv1079027675:85 , srv1223181026:85 , srv1225593425:85 , srv123001013:85 , srv1315032143:85 , srv1340216326:85 , srv1404644909:85 , srv1406705734:85 , srv1583622882:85 , srv1600378281:85 , srv1601729019:85 , srv1601819663:85 , srv1604303487:85 , srv165017599:85 , srv1711333122:85 , srv1720831510:85 , srv1735637933:85 , srv175426936:85 , srv1789951166:85 , srv1900460836:85 , srv1958157770:85 , srv1970445482:85 , srv2096312348:85 , srv219915955:85 , srv265509165:85 , srv301160101:85 , srv311844507:85 , srv370373565:85 , srv437912625:85 , srv468957009:85 , srv4864046:85 , srv494786636:85 , srv519560432:85 , srv532093445:85 , srv535539703:85 , srv546187120:85 , srv577312003:85 , srv965651555:85 , srv989811155:85 , srv1122749591:86 , srv1138839217:86 , srv1232447958:86 , srv1477555204:86 , srv1686049806:86 , srv171685076:86 , srv207163059:86 , srv725928437:86 , srv22462622:633 }
2023-11-21 17:02:51,472 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-21 17:02:51,478 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1404644909=21, srv1903621326=58, srv989811155=98, srv123001013=12, srv1406705734=22, srv782761002=92, srv1079027675=2, srv1447051999=26, srv763524426=91, srv2049073990=64, srv494786636=80, srv1583622882=30, srv140919180=24, srv646468449=87, srv1177084601=8, srv1601819663=35, srv1484766444=29, srv1966915598=62, srv1970445482=63, srv712387546=88, srv1939058585=59, srv101776415=0, srv946314210=95, srv1636917910=38, srv990037205=99, srv2075933200=66, srv1340216326=17, srv1948884955=60, srv1138839217=6, srv519560432=81, srv1847090469=55, srv1827465579=54, srv577312003=86, srv1599246014=32, srv311844507=73, srv1302008874=15, srv1223181026=10, srv1020947857=1, srv1761196331=50, srv1440185004=25, srv1600378281=33, srv1315032143=16, srv1409093035=23, srv1584598494=31, srv175426936=49, srv1122749591=4, srv1720831510=46, srv1167405475=7, srv1137142874=5, srv1689166362=43, srv207163059=65, srv1789951166=52, srv1369599019=19, srv1225593425=11, srv1899382391=56, srv1735637933=48, srv22462622=70, srv1204915642=9, srv1727187108=47, srv1819939201=53, srv1769172313=51, srv98771912=97, srv1447392286=27, srv1900460836=57, srv367987624=74, srv468957009=78, srv1477555204=28, srv906482407=93, srv546187120=84, srv735410450=90, srv1291005463=14, srv2096312348=67, srv1232447958=13, srv1601729019=34, srv965651555=96, srv1091522098=3, srv1632807692=37, srv171685076=45, srv535539703=83, srv265509165=71, srv572177968=85, srv1711333122=44, srv930561149=94, srv532093445=82, srv1686049806=41, srv211289442=68, srv301160101=72, srv1958157770=61, srv219915955=69, srv1394375565=20, srv166904965=40, srv1342524126=18, srv437912625=76, srv370373565=75, srv725928437=89, srv165017599=39, srv464336562=77, srv1688046808=42, srv1604303487=36, srv4864046=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-21 17:02:51,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-21 17:02:51,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-21 17:02:51,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-21 17:02:51,491 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-21 17:02:51,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-21 17:02:51,493 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-21 17:02:51,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-21 17:02:51,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-21 17:02:51,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-21 17:02:51,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-21 17:02:51,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-21 17:02:51,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-21 17:02:51,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-21 17:02:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-21 17:02:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-21 17:02:51,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-21 17:02:51,685 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.804543993780202E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:51,689 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv464336562:82 , srv140919180:83 , srv1447051999:83 , srv2075933200:83 , srv101776415:84 , srv1091522098:84 , srv1137142874:84 , srv1167405475:84 , srv1177084601:84 , srv1204915642:84 , srv1291005463:84 , srv1302008874:84 , srv1342524126:84 , srv1369599019:84 , srv1394375565:84 , srv1409093035:84 , srv1440185004:84 , srv1447392286:84 , srv1484766444:84 , srv1584598494:84 , srv1599246014:84 , srv1632807692:84 , srv1636917910:84 , srv166904965:84 , srv1688046808:84 , srv1689166362:84 , srv1727187108:84 , srv1761196331:84 , srv1769172313:84 , srv1819939201:84 , srv1827465579:84 , srv1847090469:84 , srv1899382391:84 , srv1903621326:84 , srv1939058585:84 , srv1948884955:84 , srv1966915598:84 , srv2049073990:84 , srv211289442:84 , srv367987624:84 , srv572177968:84 , srv646468449:84 , srv712387546:84 , srv735410450:84 , srv763524426:84 , srv782761002:84 , srv906482407:84 , srv930561149:84 , srv946314210:84 , srv98771912:84 , srv990037205:84 , srv1020947857:85 , srv1079027675:85 , srv1223181026:85 , srv1225593425:85 , srv123001013:85 , srv1315032143:85 , srv1340216326:85 , srv1404644909:85 , srv1406705734:85 , srv1583622882:85 , srv1600378281:85 , srv1601729019:85 , srv1601819663:85 , srv1604303487:85 , srv165017599:85 , srv1711333122:85 , srv1720831510:85 , srv1735637933:85 , srv175426936:85 , srv1789951166:85 , srv1900460836:85 , srv1958157770:85 , srv1970445482:85 , srv2096312348:85 , srv219915955:85 , srv265509165:85 , srv301160101:85 , srv311844507:85 , srv370373565:85 , srv437912625:85 , srv468957009:85 , srv4864046:85 , srv494786636:85 , srv519560432:85 , srv532093445:85 , srv535539703:85 , srv546187120:85 , srv577312003:85 , srv965651555:85 , srv989811155:85 , srv1122749591:86 , srv1138839217:86 , srv1232447958:86 , srv1477555204:86 , srv1686049806:86 , srv171685076:86 , srv207163059:86 , srv725928437:86 , srv22462622:633 }
2023-11-21 17:02:51,756 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=639 (was 639), ProcessCount=164 (was 167), AvailableMemoryMB=6578 (was 7064)
2023-11-21 17:02:51,774 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=639, ProcessCount=164, AvailableMemoryMB=6578
2023-11-21 17:02:51,775 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-21 17:02:51,775 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-21 17:02:51,787 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004878582:84 , srv1030757599:84 , srv1125546674:84 , srv1258779024:84 , srv1554094297:84 , srv1617856112:84 , srv1641415008:84 , srv1648294541:84 , srv1662371183:84 , srv1865271554:84 , srv1899623026:84 , srv1964891343:84 , srv1990336030:84 , srv2128863423:84 , srv2144271149:84 , srv230505801:84 , srv336962426:84 , srv39603864:84 , srv404104086:84 , srv40514067:84 , srv410648030:84 , srv48121711:84 , srv549828766:84 , srv623048681:84 , srv648610106:84 , srv713307127:84 , srv793896246:84 , srv870516939:84 , srv872380496:84 , srv1659051472:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-21 17:02:51,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-21 17:02:51,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv336962426=17, srv870516939=28, srv872380496=29, srv1125546674=2, srv1964891343=12, srv404104086=19, srv793896246=27, srv1990336030=13, srv39603864=18, srv549828766=23, srv410648030=21, srv1641415008=6, srv713307127=26, srv648610106=25, srv40514067=20, srv1899623026=11, srv2128863423=14, srv1617856112=5, srv1004878582=0, srv1030757599=1, srv2144271149=15, srv1865271554=10, srv230505801=16, srv48121711=22, srv1662371183=9, srv1648294541=7, srv1659051472=8, srv1258779024=3, srv1554094297=4, srv623048681=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-21 17:02:51,817 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-21 17:02:51,818 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-21 17:02:51,819 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-21 17:02:51,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-21 17:02:51,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-21 17:02:51,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-21 17:02:51,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-21 17:02:51,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-21 17:02:51,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-21 17:02:51,863 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0951723321268507, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:02:51,863 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.991249292817515, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0951723321268507, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-21 17:04:51,840 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9549530 different iterations.  Found a solution that moves 1837 regions; Going from a computed imbalance of 0.991249292817515 to a new imbalance of 4.2876496417733574E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6803703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:04:51,845 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1004878582:90 , srv1030757599:90 , srv1125546674:90 , srv1258779024:90 , srv1554094297:90 , srv1617856112:90 , srv1641415008:90 , srv1648294541:90 , srv1659051472:90 , srv1662371183:90 , srv1865271554:90 , srv1899623026:90 , srv1964891343:90 , srv1990336030:90 , srv2128863423:90 , srv2144271149:90 , srv230505801:90 , srv336962426:90 , srv39603864:90 , srv404104086:90 , srv40514067:90 , srv410648030:90 , srv48121711:90 , srv549828766:90 , srv623048681:90 , srv648610106:90 , srv713307127:90 , srv793896246:90 , srv870516939:90 , srv872380496:90 }
2023-11-21 17:04:51,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-21 17:04:51,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv336962426=17, srv870516939=28, srv872380496=29, srv1125546674=2, srv1964891343=12, srv404104086=19, srv793896246=27, srv1990336030=13, srv39603864=18, srv549828766=23, srv410648030=21, srv1641415008=6, srv713307127=26, srv648610106=25, srv40514067=20, srv1899623026=11, srv2128863423=14, srv1617856112=5, srv1004878582=0, srv1030757599=1, srv2144271149=15, srv1865271554=10, srv230505801=16, srv48121711=22, srv1662371183=9, srv1648294541=7, srv1659051472=8, srv1258779024=3, srv1554094297=4, srv623048681=24} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-21 17:04:51,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-21 17:04:51,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-21 17:04:51,860 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-21 17:04:51,860 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1004878582:90 , srv1030757599:90 , srv1125546674:90 , srv1258779024:90 , srv1554094297:90 , srv1617856112:90 , srv1641415008:90 , srv1648294541:90 , srv1659051472:90 , srv1662371183:90 , srv1865271554:90 , srv1899623026:90 , srv1964891343:90 , srv1990336030:90 , srv2128863423:90 , srv2144271149:90 , srv230505801:90 , srv336962426:90 , srv39603864:90 , srv404104086:90 , srv40514067:90 , srv410648030:90 , srv48121711:90 , srv549828766:90 , srv623048681:90 , srv648610106:90 , srv713307127:90 , srv793896246:90 , srv870516939:90 , srv872380496:90 }
2023-11-21 17:04:51,870 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=240 (was 639), ProcessCount=165 (was 164) - ProcessCount LEAK? -, AvailableMemoryMB=6505 (was 6578)