2023-05-21 07:53:23,300 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-21 07:53:23,617 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-21 07:53:23,727 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-21 07:53:23,787 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-21 07:53:23,795 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-21 07:53:23,809 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=171, AvailableMemoryMB=5984
2023-05-21 07:53:23,811 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-21 07:53:23,812 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-21 07:53:23,959 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1063491078:84 , srv1077365694:84 , srv1087264740:84 , srv1129784713:84 , srv1159034840:84 , srv1162007514:84 , srv1164267157:84 , srv118049146:84 , srv1183449286:84 , srv121217093:84 , srv1224216274:84 , srv1261089087:84 , srv1265504603:84 , srv1333677100:84 , srv1335029551:84 , srv1339754776:84 , srv1385317108:84 , srv1441653368:84 , srv1443043597:84 , srv1452923614:84 , srv1470459412:84 , srv147696192:84 , srv1523842994:84 , srv1552453679:84 , srv1566687216:84 , srv1603744859:84 , srv1615942263:84 , srv1629286184:84 , srv163072640:84 , srv16404158:84 , srv1650356900:84 , srv1659816224:84 , srv1679743317:84 , srv16897976:84 , srv1697360353:84 , srv1728084373:84 , srv1734114438:84 , srv174975758:84 , srv1760347724:84 , srv1783222486:84 , srv1792497804:84 , srv180231523:84 , srv1816914039:84 , srv1826730074:84 , srv1855217724:84 , srv1863494198:84 , srv1936907326:84 , srv1943380612:84 , srv1986418519:84 , srv2064120898:84 , srv2067540594:84 , srv2105789629:84 , srv2133521520:84 , srv2144606458:84 , srv220330812:84 , srv23057805:84 , srv270482090:84 , srv283628050:84 , srv299206095:84 , srv333820378:84 , srv338530630:84 , srv352097514:84 , srv402586017:84 , srv403367217:84 , srv410117362:84 , srv4185837:84 , srv451963661:84 , srv461176151:84 , srv462075856:84 , srv463314613:84 , srv469165784:84 , srv483646321:84 , srv515622669:84 , srv522218858:84 , srv536784429:84 , srv550395449:84 , srv558024634:84 , srv608612094:84 , srv625283812:84 , srv634474926:84 , srv645443995:84 , srv689973448:84 , srv696215236:84 , srv714118542:84 , srv75440456:84 , srv765290818:84 , srv796835808:84 , srv797777197:84 , srv804076046:84 , srv823666773:84 , srv828426719:84 , srv840372512:84 , srv844767965:84 , srv851225075:84 , srv910590056:84 , srv917088506:84 , srv934664228:84 , srv985950611:84 , srv998718053:84 , srv1488286609:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-21 07:53:23,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 07:53:23,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1063491078=0, srv1863494198=46, srv462075856=69, srv180231523=42, srv463314613=70, srv1855217724=45, srv804076046=89, srv402586017=63, srv1566687216=25, srv1488286609=22, srv840372512=92, srv714118542=84, srv1936907326=47, srv634474926=80, srv1697360353=35, srv1734114438=37, srv1162007514=5, srv1183449286=8, srv1441653368=17, srv823666773=90, srv174975758=38, srv2064120898=50, srv1629286184=28, srv696215236=83, srv1159034840=4, srv1261089087=11, srv1333677100=13, srv608612094=78, srv121217093=9, srv1552453679=24, srv1943380612=48, srv1443043597=18, srv1470459412=20, srv1728084373=36, srv934664228=97, srv1760347724=39, srv1792497804=41, srv1603744859=26, srv23057805=56, srv645443995=81, srv1129784713=3, srv1659816224=32, srv828426719=91, srv16897976=34, srv515622669=73, srv118049146=7, srv1783222486=40, srv536784429=75, srv522218858=74, srv765290818=86, srv2105789629=52, srv2133521520=53, srv558024634=77, srv1385317108=16, srv1523842994=23, srv220330812=55, srv403367217=64, srv998718053=99, srv147696192=21, srv4185837=66, srv2067540594=51, srv299206095=59, srv1452923614=19, srv1816914039=43, srv1077365694=1, srv410117362=65, srv163072640=29, srv16404158=30, srv2144606458=54, srv1650356900=31, srv1335029551=14, srv1339754776=15, srv469165784=71, srv1986418519=49, srv625283812=79, srv796835808=87, srv1615942263=27, srv844767965=93, srv75440456=85, srv985950611=98, srv797777197=88, srv1265504603=12, srv1224216274=10, srv283628050=58, srv333820378=60, srv270482090=57, srv451963661=67, srv352097514=62, srv461176151=68, srv851225075=94, srv1164267157=6, srv1826730074=44, srv910590056=95, srv1679743317=33, srv338530630=61, srv483646321=72, srv550395449=76, srv689973448=82, srv917088506=96, srv1087264740=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 07:53:24,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 07:53:24,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 07:53:24,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 07:53:24,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 07:53:24,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 07:53:24,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 07:53:24,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 07:53:24,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 07:53:24,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 07:53:24,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 07:53:24,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 07:53:24,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 07:53:24,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 07:53:24,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 07:53:24,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 07:53:24,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 07:53:24,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 07:53:24,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 07:53:24,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 07:53:24,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 07:53:24,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 07:53:24,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 07:53:24,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 07:53:24,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 07:53:24,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 07:53:24,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 07:53:24,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 07:53:24,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 07:53:24,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 07:53:24,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 07:53:24,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 07:53:24,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 07:53:24,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 07:53:24,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 07:53:24,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 07:53:24,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 07:53:24,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 07:53:24,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 07:53:24,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 07:53:24,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 07:53:24,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 07:53:24,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 07:53:24,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 07:53:24,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 07:53:24,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 07:53:24,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 07:53:24,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 07:53:24,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 07:53:24,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 07:53:24,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 07:53:24,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 07:53:24,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 07:53:24,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 07:53:24,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 07:53:24,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 07:53:24,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 07:53:24,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 07:53:24,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 07:53:24,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 07:53:24,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 07:53:24,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 07:53:24,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 07:53:24,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 07:53:24,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 07:53:24,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 07:53:24,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 07:53:24,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 07:53:24,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 07:53:24,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 07:53:24,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 07:53:24,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 07:53:24,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 07:53:24,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 07:53:24,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 07:53:24,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 07:53:24,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 07:53:24,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 07:53:24,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 07:53:24,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 07:53:24,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 07:53:24,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 07:53:24,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 07:53:24,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 07:53:24,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 07:53:24,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 07:53:24,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 07:53:24,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 07:53:24,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 07:53:24,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 07:53:24,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 07:53:24,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 07:53:24,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 07:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 07:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 07:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 07:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 07:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 07:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 07:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 07:53:24,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 07:53:24,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 07:53:24,389 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:24,389 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-21 07:53:24,390 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-21 07:53:24,786 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 666 ms to try 5000 different iterations.  Found a solution that moves 4719 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1816698104272287. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03992479524644749, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0569482014035355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5243333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03992479524644749, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24758333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:24,831 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1164267157:82 , srv1087264740:83 , srv1523842994:83 , srv2064120898:83 , srv840372512:83 , srv917088506:83 , srv1077365694:84 , srv1159034840:84 , srv1162007514:84 , srv121217093:84 , srv163072640:84 , srv1659816224:84 , srv283628050:84 , srv625283812:84 , srv765290818:84 , srv1063491078:85 , srv1265504603:85 , srv1552453679:85 , srv1566687216:85 , srv1679743317:85 , srv1734114438:85 , srv180231523:85 , srv1816914039:85 , srv1826730074:85 , srv1936907326:85 , srv2144606458:85 , srv338530630:85 , srv402586017:85 , srv483646321:85 , srv536784429:85 , srv558024634:85 , srv608612094:85 , srv689973448:85 , srv797777197:85 , srv828426719:85 , srv844767965:85 , srv985950611:85 , srv998718053:85 , srv1335029551:86 , srv1339754776:86 , srv1385317108:86 , srv1615942263:86 , srv1650356900:86 , srv1697360353:86 , srv1728084373:86 , srv1783222486:86 , srv1792497804:86 , srv1943380612:86 , srv2133521520:86 , srv270482090:86 , srv333820378:86 , srv4185837:86 , srv451963661:86 , srv462075856:86 , srv515622669:86 , srv645443995:86 , srv75440456:86 , srv934664228:86 , srv118049146:87 , srv1333677100:87 , srv1441653368:87 , srv1603744859:87 , srv16404158:87 , srv1986418519:87 , srv2067540594:87 , srv220330812:87 , srv23057805:87 , srv403367217:87 , srv461176151:87 , srv696215236:87 , srv796835808:87 , srv804076046:87 , srv1129784713:88 , srv1443043597:88 , srv1452923614:88 , srv1629286184:88 , srv174975758:88 , srv1760347724:88 , srv2105789629:88 , srv299206095:88 , srv410117362:88 , srv469165784:88 , srv634474926:88 , srv714118542:88 , srv823666773:88 , srv851225075:88 , srv910590056:88 , srv1261089087:89 , srv1470459412:89 , srv147696192:89 , srv1863494198:89 , srv522218858:89 , srv550395449:89 , srv1183449286:90 , srv1224216274:90 , srv1855217724:90 , srv16897976:94 , srv352097514:95 , srv463314613:96 , srv1488286609:445 }
2023-05-21 07:53:24,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 07:53:24,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1063491078=0, srv1863494198=46, srv462075856=69, srv180231523=42, srv463314613=70, srv1855217724=45, srv804076046=89, srv402586017=63, srv1566687216=25, srv1488286609=22, srv840372512=92, srv714118542=84, srv1936907326=47, srv634474926=80, srv1697360353=35, srv1734114438=37, srv1162007514=5, srv1183449286=8, srv1441653368=17, srv823666773=90, srv174975758=38, srv2064120898=50, srv1629286184=28, srv696215236=83, srv1159034840=4, srv1261089087=11, srv1333677100=13, srv608612094=78, srv121217093=9, srv1552453679=24, srv1943380612=48, srv1443043597=18, srv1470459412=20, srv1728084373=36, srv934664228=97, srv1760347724=39, srv1792497804=41, srv1603744859=26, srv23057805=56, srv645443995=81, srv1129784713=3, srv1659816224=32, srv828426719=91, srv16897976=34, srv515622669=73, srv118049146=7, srv1783222486=40, srv536784429=75, srv522218858=74, srv765290818=86, srv2105789629=52, srv2133521520=53, srv558024634=77, srv1385317108=16, srv1523842994=23, srv220330812=55, srv403367217=64, srv998718053=99, srv147696192=21, srv4185837=66, srv2067540594=51, srv299206095=59, srv1452923614=19, srv1816914039=43, srv1077365694=1, srv410117362=65, srv163072640=29, srv16404158=30, srv2144606458=54, srv1650356900=31, srv1335029551=14, srv1339754776=15, srv469165784=71, srv1986418519=49, srv625283812=79, srv796835808=87, srv1615942263=27, srv844767965=93, srv75440456=85, srv985950611=98, srv797777197=88, srv1265504603=12, srv1224216274=10, srv283628050=58, srv333820378=60, srv270482090=57, srv451963661=67, srv352097514=62, srv461176151=68, srv851225075=94, srv1164267157=6, srv1826730074=44, srv910590056=95, srv1679743317=33, srv338530630=61, srv483646321=72, srv550395449=76, srv689973448=82, srv917088506=96, srv1087264740=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 07:53:24,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 07:53:24,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 07:53:24,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 07:53:24,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 07:53:24,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 07:53:24,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 07:53:24,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 07:53:24,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 07:53:24,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 07:53:24,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 07:53:24,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 07:53:24,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 07:53:24,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 07:53:24,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 07:53:24,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 07:53:24,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 07:53:24,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 07:53:24,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 07:53:24,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 07:53:24,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 07:53:24,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 07:53:24,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 07:53:24,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 07:53:24,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 07:53:24,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 07:53:24,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 07:53:24,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 07:53:24,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 07:53:24,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 07:53:24,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 07:53:24,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 07:53:24,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 07:53:24,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 07:53:24,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 07:53:24,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 07:53:24,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 07:53:24,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 07:53:24,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 07:53:24,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 07:53:24,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 07:53:24,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 07:53:24,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 07:53:24,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 07:53:24,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 07:53:24,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 07:53:24,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 07:53:24,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 07:53:24,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 07:53:24,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 07:53:24,868 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 07:53:24,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 07:53:24,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 07:53:24,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 07:53:24,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 07:53:24,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 07:53:24,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 07:53:24,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 07:53:24,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 07:53:25,089 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03992479524644749, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0569482014035355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03992479524644749, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24758333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:25,090 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-21 07:53:25,090 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18163676728298342, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03992479524644749, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0569482014035355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03992479524644749, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1765, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24758333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-21 07:53:25,514 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 624 ms to try 5000 different iterations.  Found a solution that moves 2643 regions; Going from a computed imbalance of 0.18163676728298342 to a new imbalance of 0.015226942774371437. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020164555608929227); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032038837262824624, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2936666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020164555608929227); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:25,528 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv16897976:76 , srv463314613:78 , srv352097514:79 , srv1183449286:82 , srv1629286184:82 , srv1760347724:82 , srv550395449:82 , srv1452923614:83 , srv147696192:83 , srv1863494198:83 , srv2067540594:83 , srv2105789629:83 , srv634474926:83 , srv1129784713:84 , srv1224216274:84 , srv1261089087:84 , srv1335029551:84 , srv1443043597:84 , srv1470459412:84 , srv16404158:84 , srv174975758:84 , srv1826730074:84 , srv1855217724:84 , srv299206095:84 , srv403367217:84 , srv461176151:84 , srv645443995:84 , srv910590056:84 , srv118049146:85 , srv1265504603:85 , srv1339754776:85 , srv1441653368:85 , srv1552453679:85 , srv1603744859:85 , srv1697360353:85 , srv1734114438:85 , srv180231523:85 , srv220330812:85 , srv23057805:85 , srv270482090:85 , srv410117362:85 , srv4185837:85 , srv558024634:85 , srv714118542:85 , srv796835808:85 , srv804076046:85 , srv1333677100:86 , srv1679743317:86 , srv1728084373:86 , srv1816914039:86 , srv1936907326:86 , srv1943380612:86 , srv1986418519:86 , srv2144606458:86 , srv469165784:86 , srv515622669:86 , srv522218858:86 , srv536784429:86 , srv823666773:86 , srv1063491078:87 , srv1159034840:87 , srv121217093:87 , srv1385317108:87 , srv1615942263:87 , srv1659816224:87 , srv1783222486:87 , srv2133521520:87 , srv283628050:87 , srv462075856:87 , srv608612094:87 , srv689973448:87 , srv696215236:87 , srv844767965:87 , srv1077365694:88 , srv1087264740:88 , srv1162007514:88 , srv1164267157:88 , srv1523842994:88 , srv1566687216:88 , srv1650356900:88 , srv338530630:88 , srv402586017:88 , srv451963661:88 , srv75440456:88 , srv765290818:88 , srv797777197:88 , srv828426719:88 , srv840372512:88 , srv851225075:88 , srv934664228:88 , srv985950611:88 , srv163072640:89 , srv1792497804:89 , srv625283812:89 , srv333820378:90 , srv483646321:90 , srv998718053:90 , srv2064120898:92 , srv917088506:94 , srv1488286609:508 }
2023-05-21 07:53:25,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 07:53:25,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1063491078=0, srv1863494198=46, srv462075856=69, srv180231523=42, srv463314613=70, srv1855217724=45, srv804076046=89, srv402586017=63, srv1566687216=25, srv1488286609=22, srv840372512=92, srv714118542=84, srv1936907326=47, srv634474926=80, srv1697360353=35, srv1734114438=37, srv1162007514=5, srv1183449286=8, srv1441653368=17, srv823666773=90, srv174975758=38, srv2064120898=50, srv1629286184=28, srv696215236=83, srv1159034840=4, srv1261089087=11, srv1333677100=13, srv608612094=78, srv121217093=9, srv1552453679=24, srv1943380612=48, srv1443043597=18, srv1470459412=20, srv1728084373=36, srv934664228=97, srv1760347724=39, srv1792497804=41, srv1603744859=26, srv23057805=56, srv645443995=81, srv1129784713=3, srv1659816224=32, srv828426719=91, srv16897976=34, srv515622669=73, srv118049146=7, srv1783222486=40, srv536784429=75, srv522218858=74, srv765290818=86, srv2105789629=52, srv2133521520=53, srv558024634=77, srv1385317108=16, srv1523842994=23, srv220330812=55, srv403367217=64, srv998718053=99, srv147696192=21, srv4185837=66, srv2067540594=51, srv299206095=59, srv1452923614=19, srv1816914039=43, srv1077365694=1, srv410117362=65, srv163072640=29, srv16404158=30, srv2144606458=54, srv1650356900=31, srv1335029551=14, srv1339754776=15, srv469165784=71, srv1986418519=49, srv625283812=79, srv796835808=87, srv1615942263=27, srv844767965=93, srv75440456=85, srv985950611=98, srv797777197=88, srv1265504603=12, srv1224216274=10, srv283628050=58, srv333820378=60, srv270482090=57, srv451963661=67, srv352097514=62, srv461176151=68, srv851225075=94, srv1164267157=6, srv1826730074=44, srv910590056=95, srv1679743317=33, srv338530630=61, srv483646321=72, srv550395449=76, srv689973448=82, srv917088506=96, srv1087264740=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 07:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 07:53:25,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 07:53:25,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 07:53:25,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 07:53:25,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 07:53:25,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 07:53:25,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 07:53:25,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 07:53:25,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 07:53:25,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 07:53:25,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 07:53:25,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 07:53:25,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 07:53:25,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 07:53:25,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 07:53:25,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 07:53:25,718 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020164555608929227); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032038837262824624, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020164555608929227); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:25,718 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-21 07:53:25,719 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015208436092820202, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020164555608929227); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.032038837262824624, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020164555608929227); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04791666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-21 07:53:26,068 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 467 ms to try 5000 different iterations.  Found a solution that moves 1021 regions; Going from a computed imbalance of 0.015208436092820202 to a new imbalance of 5.510346637649907E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007124216319137535); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009870087677015036); MoveCostFunction : (multiplier=7.0, imbalance=0.11344444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007124216319137535); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:26,069 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv333820378:81 , srv851225075:81 , srv917088506:81 , srv451963661:82 , srv522218858:82 , srv998718053:82 , srv1261089087:83 , srv1615942263:83 , srv163072640:83 , srv1792497804:83 , srv1855217724:83 , srv463314613:83 , srv483646321:83 , srv696215236:83 , srv75440456:83 , srv1129784713:84 , srv1385317108:84 , srv1566687216:84 , srv1650356900:84 , srv2064120898:84 , srv299206095:84 , srv338530630:84 , srv402586017:84 , srv469165784:84 , srv714118542:84 , srv828426719:84 , srv934664228:84 , srv985950611:84 , srv1162007514:85 , srv118049146:85 , srv1183449286:85 , srv1224216274:85 , srv1441653368:85 , srv1470459412:85 , srv1603744859:85 , srv174975758:85 , srv1816914039:85 , srv1863494198:85 , srv1986418519:85 , srv2133521520:85 , srv23057805:85 , srv352097514:85 , srv410117362:85 , srv462075856:85 , srv515622669:85 , srv536784429:85 , srv608612094:85 , srv625283812:85 , srv689973448:85 , srv796835808:85 , srv797777197:85 , srv804076046:85 , srv823666773:85 , srv910590056:85 , srv1063491078:86 , srv1077365694:86 , srv1087264740:86 , srv1164267157:86 , srv121217093:86 , srv1333677100:86 , srv1443043597:86 , srv1452923614:86 , srv147696192:86 , srv1523842994:86 , srv16404158:86 , srv1659816224:86 , srv1697360353:86 , srv1728084373:86 , srv1783222486:86 , srv180231523:86 , srv1936907326:86 , srv1943380612:86 , srv2067540594:86 , srv2144606458:86 , srv220330812:86 , srv283628050:86 , srv4185837:86 , srv461176151:86 , srv765290818:86 , srv840372512:86 , srv1159034840:87 , srv1265504603:87 , srv1335029551:87 , srv1339754776:87 , srv1679743317:87 , srv16897976:87 , srv1734114438:87 , srv1760347724:87 , srv270482090:87 , srv403367217:87 , srv550395449:87 , srv558024634:87 , srv634474926:87 , srv844767965:87 , srv1552453679:88 , srv1629286184:88 , srv1826730074:88 , srv2105789629:88 , srv645443995:88 , srv1488286609:568 }
2023-05-21 07:53:26,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 07:53:26,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1063491078=0, srv1863494198=46, srv462075856=69, srv180231523=42, srv463314613=70, srv1855217724=45, srv804076046=89, srv402586017=63, srv1566687216=25, srv1488286609=22, srv840372512=92, srv714118542=84, srv1936907326=47, srv634474926=80, srv1697360353=35, srv1734114438=37, srv1162007514=5, srv1183449286=8, srv1441653368=17, srv823666773=90, srv174975758=38, srv2064120898=50, srv1629286184=28, srv696215236=83, srv1159034840=4, srv1261089087=11, srv1333677100=13, srv608612094=78, srv121217093=9, srv1552453679=24, srv1943380612=48, srv1443043597=18, srv1470459412=20, srv1728084373=36, srv934664228=97, srv1760347724=39, srv1792497804=41, srv1603744859=26, srv23057805=56, srv645443995=81, srv1129784713=3, srv1659816224=32, srv828426719=91, srv16897976=34, srv515622669=73, srv118049146=7, srv1783222486=40, srv536784429=75, srv522218858=74, srv765290818=86, srv2105789629=52, srv2133521520=53, srv558024634=77, srv1385317108=16, srv1523842994=23, srv220330812=55, srv403367217=64, srv998718053=99, srv147696192=21, srv4185837=66, srv2067540594=51, srv299206095=59, srv1452923614=19, srv1816914039=43, srv1077365694=1, srv410117362=65, srv163072640=29, srv16404158=30, srv2144606458=54, srv1650356900=31, srv1335029551=14, srv1339754776=15, srv469165784=71, srv1986418519=49, srv625283812=79, srv796835808=87, srv1615942263=27, srv844767965=93, srv75440456=85, srv985950611=98, srv797777197=88, srv1265504603=12, srv1224216274=10, srv283628050=58, srv333820378=60, srv270482090=57, srv451963661=67, srv352097514=62, srv461176151=68, srv851225075=94, srv1164267157=6, srv1826730074=44, srv910590056=95, srv1679743317=33, srv338530630=61, srv483646321=72, srv550395449=76, srv689973448=82, srv917088506=96, srv1087264740=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 07:53:26,085 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 07:53:26,086 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 07:53:26,087 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 07:53:26,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 07:53:26,089 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 07:53:26,090 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 07:53:26,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 07:53:26,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007124216319137535); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009870087677015036); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007124216319137535); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:26,195 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-21 07:53:26,195 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.438854689621863E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007124216319137535); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009870087677015036); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007124216319137535); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004333333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-21 07:53:26,544 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 445 ms to try 5000 different iterations.  Found a solution that moves 287 regions; Going from a computed imbalance of 5.438854689621863E-4 to a new imbalance of 2.3715779092836363E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.03188888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:26,545 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv823666773:82 , srv1224216274:83 , srv1333677100:83 , srv1783222486:83 , srv2064120898:83 , srv2105789629:83 , srv352097514:83 , srv765290818:83 , srv1063491078:84 , srv1077365694:84 , srv1339754776:84 , srv1443043597:84 , srv1470459412:84 , srv147696192:84 , srv1552453679:84 , srv1629286184:84 , srv16404158:84 , srv1650356900:84 , srv1659816224:84 , srv1728084373:84 , srv174975758:84 , srv1792497804:84 , srv1943380612:84 , srv1986418519:84 , srv2133521520:84 , srv220330812:84 , srv270482090:84 , srv403367217:84 , srv4185837:84 , srv469165784:84 , srv483646321:84 , srv522218858:84 , srv550395449:84 , srv625283812:84 , srv634474926:84 , srv645443995:84 , srv689973448:84 , srv797777197:84 , srv844767965:84 , srv851225075:84 , srv917088506:84 , srv934664228:84 , srv985950611:84 , srv1087264740:85 , srv1159034840:85 , srv1162007514:85 , srv118049146:85 , srv1183449286:85 , srv121217093:85 , srv1265504603:85 , srv1385317108:85 , srv1441653368:85 , srv1452923614:85 , srv1523842994:85 , srv1603744859:85 , srv1679743317:85 , srv16897976:85 , srv1697360353:85 , srv1734114438:85 , srv180231523:85 , srv1863494198:85 , srv1936907326:85 , srv2144606458:85 , srv23057805:85 , srv283628050:85 , srv333820378:85 , srv338530630:85 , srv402586017:85 , srv410117362:85 , srv461176151:85 , srv462075856:85 , srv463314613:85 , srv515622669:85 , srv558024634:85 , srv608612094:85 , srv696215236:85 , srv714118542:85 , srv796835808:85 , srv804076046:85 , srv828426719:85 , srv840372512:85 , srv910590056:85 , srv998718053:85 , srv1129784713:86 , srv1164267157:86 , srv1261089087:86 , srv1335029551:86 , srv1566687216:86 , srv1615942263:86 , srv163072640:86 , srv1760347724:86 , srv1816914039:86 , srv1826730074:86 , srv1855217724:86 , srv2067540594:86 , srv299206095:86 , srv451963661:86 , srv536784429:86 , srv75440456:86 , srv1488286609:621 }
2023-05-21 07:53:26,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 07:53:26,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1063491078=0, srv1863494198=46, srv462075856=69, srv180231523=42, srv463314613=70, srv1855217724=45, srv804076046=89, srv402586017=63, srv1566687216=25, srv1488286609=22, srv840372512=92, srv714118542=84, srv1936907326=47, srv634474926=80, srv1697360353=35, srv1734114438=37, srv1162007514=5, srv1183449286=8, srv1441653368=17, srv823666773=90, srv174975758=38, srv2064120898=50, srv1629286184=28, srv696215236=83, srv1159034840=4, srv1261089087=11, srv1333677100=13, srv608612094=78, srv121217093=9, srv1552453679=24, srv1943380612=48, srv1443043597=18, srv1470459412=20, srv1728084373=36, srv934664228=97, srv1760347724=39, srv1792497804=41, srv1603744859=26, srv23057805=56, srv645443995=81, srv1129784713=3, srv1659816224=32, srv828426719=91, srv16897976=34, srv515622669=73, srv118049146=7, srv1783222486=40, srv536784429=75, srv522218858=74, srv765290818=86, srv2105789629=52, srv2133521520=53, srv558024634=77, srv1385317108=16, srv1523842994=23, srv220330812=55, srv403367217=64, srv998718053=99, srv147696192=21, srv4185837=66, srv2067540594=51, srv299206095=59, srv1452923614=19, srv1816914039=43, srv1077365694=1, srv410117362=65, srv163072640=29, srv16404158=30, srv2144606458=54, srv1650356900=31, srv1335029551=14, srv1339754776=15, srv469165784=71, srv1986418519=49, srv625283812=79, srv796835808=87, srv1615942263=27, srv844767965=93, srv75440456=85, srv985950611=98, srv797777197=88, srv1265504603=12, srv1224216274=10, srv283628050=58, srv333820378=60, srv270482090=57, srv451963661=67, srv352097514=62, srv461176151=68, srv851225075=94, srv1164267157=6, srv1826730074=44, srv910590056=95, srv1679743317=33, srv338530630=61, srv483646321=72, srv550395449=76, srv689973448=82, srv917088506=96, srv1087264740=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 07:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 07:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 07:53:26,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 07:53:26,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 07:53:26,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-21 07:53:26,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-21 07:53:26,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-21 07:53:26,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-21 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-21 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-21 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-21 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-21 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-21 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-21 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-21 07:53:26,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-21 07:53:26,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-21 07:53:26,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-21 07:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-21 07:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-21 07:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-21 07:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-21 07:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-21 07:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-21 07:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-21 07:53:26,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-21 07:53:26,677 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.1706162140431976E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00259499648053841); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:26,678 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv823666773:82 , srv1224216274:83 , srv1333677100:83 , srv1783222486:83 , srv2064120898:83 , srv2105789629:83 , srv352097514:83 , srv765290818:83 , srv1063491078:84 , srv1077365694:84 , srv1339754776:84 , srv1443043597:84 , srv1470459412:84 , srv147696192:84 , srv1552453679:84 , srv1629286184:84 , srv16404158:84 , srv1650356900:84 , srv1659816224:84 , srv1728084373:84 , srv174975758:84 , srv1792497804:84 , srv1943380612:84 , srv1986418519:84 , srv2133521520:84 , srv220330812:84 , srv270482090:84 , srv403367217:84 , srv4185837:84 , srv469165784:84 , srv483646321:84 , srv522218858:84 , srv550395449:84 , srv625283812:84 , srv634474926:84 , srv645443995:84 , srv689973448:84 , srv797777197:84 , srv844767965:84 , srv851225075:84 , srv917088506:84 , srv934664228:84 , srv985950611:84 , srv1087264740:85 , srv1159034840:85 , srv1162007514:85 , srv118049146:85 , srv1183449286:85 , srv121217093:85 , srv1265504603:85 , srv1385317108:85 , srv1441653368:85 , srv1452923614:85 , srv1523842994:85 , srv1603744859:85 , srv1679743317:85 , srv16897976:85 , srv1697360353:85 , srv1734114438:85 , srv180231523:85 , srv1863494198:85 , srv1936907326:85 , srv2144606458:85 , srv23057805:85 , srv283628050:85 , srv333820378:85 , srv338530630:85 , srv402586017:85 , srv410117362:85 , srv461176151:85 , srv462075856:85 , srv463314613:85 , srv515622669:85 , srv558024634:85 , srv608612094:85 , srv696215236:85 , srv714118542:85 , srv796835808:85 , srv804076046:85 , srv828426719:85 , srv840372512:85 , srv910590056:85 , srv998718053:85 , srv1129784713:86 , srv1164267157:86 , srv1261089087:86 , srv1335029551:86 , srv1566687216:86 , srv1615942263:86 , srv163072640:86 , srv1760347724:86 , srv1816914039:86 , srv1826730074:86 , srv1855217724:86 , srv2067540594:86 , srv299206095:86 , srv451963661:86 , srv536784429:86 , srv75440456:86 , srv1488286609:621 }
2023-05-21 07:53:26,710 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 360), ProcessCount=171 (was 171), AvailableMemoryMB=5358 (was 5984)
2023-05-21 07:53:26,717 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=171, AvailableMemoryMB=5358
2023-05-21 07:53:26,718 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-21 07:53:26,718 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-21 07:53:26,730 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1023653044:84 , srv1056183071:84 , srv1167001015:84 , srv1170352076:84 , srv1398043014:84 , srv1519901919:84 , srv1593935152:84 , srv1663054928:84 , srv1743925420:84 , srv1780958568:84 , srv1826613895:84 , srv1905857796:84 , srv1911462704:84 , srv1937286976:84 , srv2001305479:84 , srv2017884803:84 , srv2067964482:84 , srv2078554992:84 , srv2133526435:84 , srv318666218:84 , srv337058266:84 , srv3428133:84 , srv445458429:84 , srv457549471:84 , srv652091026:84 , srv673484671:84 , srv829897366:84 , srv899568794:84 , srv940519439:84 , srv1429740043:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-21 07:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 07:53:26,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1593935152=7, srv1663054928=8, srv2133526435=19, srv829897366=27, srv3428133=22, srv2001305479=15, srv673484671=26, srv1743925420=9, srv652091026=25, srv1519901919=6, srv1398043014=4, srv2078554992=18, srv2067964482=17, srv940519439=29, srv899568794=28, srv337058266=21, srv1429740043=5, srv318666218=20, srv1780958568=10, srv1056183071=1, srv1937286976=14, srv445458429=23, srv1911462704=13, srv1170352076=3, srv457549471=24, srv1023653044=0, srv1826613895=11, srv1905857796=12, srv2017884803=16, srv1167001015=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 07:53:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 07:53:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 07:53:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 07:53:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 07:53:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 07:53:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 07:53:26,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 07:53:26,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 07:53:26,747 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-21 07:53:26,771 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1781186657267804, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:53:26,771 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912754289363874, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1781186657267804, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-21 07:55:26,751 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6141502 different iterations.  Found a solution that moves 1857 regions; Going from a computed imbalance of 0.9912754289363874 to a new imbalance of 4.334330639506329E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6877777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:55:26,757 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1023653044:90 , srv1056183071:90 , srv1167001015:90 , srv1170352076:90 , srv1398043014:90 , srv1429740043:90 , srv1519901919:90 , srv1593935152:90 , srv1663054928:90 , srv1743925420:90 , srv1780958568:90 , srv1826613895:90 , srv1905857796:90 , srv1911462704:90 , srv1937286976:90 , srv2001305479:90 , srv2017884803:90 , srv2067964482:90 , srv2078554992:90 , srv2133526435:90 , srv318666218:90 , srv337058266:90 , srv3428133:90 , srv445458429:90 , srv457549471:90 , srv652091026:90 , srv673484671:90 , srv829897366:90 , srv899568794:90 , srv940519439:90 }
2023-05-21 07:55:26,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-21 07:55:26,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1593935152=7, srv1663054928=8, srv2133526435=19, srv829897366=27, srv3428133=22, srv2001305479=15, srv673484671=26, srv1743925420=9, srv652091026=25, srv1519901919=6, srv1398043014=4, srv2078554992=18, srv2067964482=17, srv940519439=29, srv899568794=28, srv337058266=21, srv1429740043=5, srv318666218=20, srv1780958568=10, srv1056183071=1, srv1937286976=14, srv445458429=23, srv1911462704=13, srv1170352076=3, srv457549471=24, srv1023653044=0, srv1826613895=11, srv1905857796=12, srv2017884803=16, srv1167001015=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-21 07:55:26,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-21 07:55:26,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-21 07:55:26,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-21 07:55:26,761 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-21 07:55:26,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-21 07:55:26,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-21 07:55:26,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-21 07:55:26,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-21 07:55:26,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-21 07:55:26,774 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-21 07:55:26,774 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1023653044:90 , srv1056183071:90 , srv1167001015:90 , srv1170352076:90 , srv1398043014:90 , srv1429740043:90 , srv1519901919:90 , srv1593935152:90 , srv1663054928:90 , srv1743925420:90 , srv1780958568:90 , srv1826613895:90 , srv1905857796:90 , srv1911462704:90 , srv1937286976:90 , srv2001305479:90 , srv2017884803:90 , srv2067964482:90 , srv2078554992:90 , srv2133526435:90 , srv318666218:90 , srv337058266:90 , srv3428133:90 , srv445458429:90 , srv457549471:90 , srv652091026:90 , srv673484671:90 , srv829897366:90 , srv899568794:90 , srv940519439:90 }
2023-05-21 07:55:26,786 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 360), ProcessCount=169 (was 171), AvailableMemoryMB=4759 (was 5358)