2023-04-17 07:53:38,365 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-04-17 07:53:38,639 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 07:53:38,744 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-17 07:53:38,795 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-17 07:53:38,801 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-17 07:53:38,813 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=316, ProcessCount=173, AvailableMemoryMB=8790 2023-04-17 07:53:38,815 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-17 07:53:38,815 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-17 07:53:38,960 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026735288:84 , srv1031610187:84 , srv1052160628:84 , srv1060178775:84 , srv1077663760:84 , srv1090510460:84 , srv1102614534:84 , srv110274250:84 , srv1113383162:84 , srv1127153572:84 , srv1151582595:84 , srv1162485188:84 , srv1168153621:84 , srv1169633589:84 , srv1174797896:84 , srv1201998408:84 , srv1205234383:84 , srv1240532509:84 , srv1266315448:84 , srv1271463110:84 , srv1275377394:84 , srv12919135:84 , srv1293597841:84 , srv1323375962:84 , srv1335021433:84 , srv1370090283:84 , srv1434196804:84 , srv1443409733:84 , srv1509622807:84 , srv1536437830:84 , srv1561009810:84 , srv1576001473:84 , srv158418878:84 , srv1611393882:84 , srv1612514869:84 , srv1633218359:84 , srv1642396739:84 , srv1682717290:84 , srv1693468680:84 , srv170372505:84 , srv1716776192:84 , srv1732743369:84 , srv1751020316:84 , srv1799970147:84 , srv1825561643:84 , srv1835408694:84 , srv1857140602:84 , srv1861509348:84 , srv1889565465:84 , srv190125593:84 , srv198441611:84 , srv2018549464:84 , srv2025639013:84 , srv2029827628:84 , srv205077331:84 , srv2064892389:84 , srv2072798377:84 , srv207842814:84 , srv210341128:84 , srv2139195670:84 , srv217146624:84 , srv219780610:84 , srv235614059:84 , srv237759486:84 , srv245008912:84 , srv278330364:84 , srv292841913:84 , srv297035347:84 , srv306988742:84 , srv32055905:84 , srv340321010:84 , srv34068987:84 , srv454132294:84 , srv459890251:84 , srv470338834:84 , srv471803156:84 , srv472559042:84 , srv473777089:84 , srv478391326:84 , srv498808830:84 , srv520338735:84 , srv525158325:84 , srv533961744:84 , srv53580214:84 , srv569147418:84 , srv615764351:84 , srv619344537:84 , srv663023662:84 , srv709485678:84 , srv737868675:84 , srv776765601:84 , srv807262758:84 , srv852733196:84 , srv870211856:84 , srv90919390:84 , srv910446437:84 , srv941440069:84 , srv948799373:84 , srv996669060:84 , srv459572118:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-04-17 07:53:38,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-17 07:53:38,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv569147418=85, srv1576001473=31, srv235614059=62, srv1031610187=1, srv1275377394=20, srv948799373=98, srv190125593=49, srv470338834=75, srv110274250=7, srv472559042=77, srv278330364=65, srv852733196=93, srv941440069=97, srv1102614534=6, srv340321010=70, srv520338735=81, srv870211856=94, srv1682717290=37, srv1201998408=15, srv1127153572=9, srv1052160628=2, srv1889565465=48, srv525158325=82, srv1799970147=43, srv1509622807=28, srv1732743369=41, srv1861509348=47, srv1271463110=19, srv1835408694=45, srv1026735288=0, srv1077663760=4, srv158418878=32, srv1090510460=5, srv1611393882=33, srv245008912=64, srv170372505=39, srv1335021433=24, srv210341128=58, srv807262758=92, srv1693468680=38, srv1151582595=10, srv498808830=80, srv615764351=86, srv217146624=60, srv1825561643=44, srv1266315448=18, srv2025639013=52, srv459890251=74, srv663023662=88, srv2139195670=59, srv219780610=61, srv1060178775=3, srv1443409733=27, srv1642396739=36, srv1716776192=40, srv1561009810=30, srv619344537=87, srv1323375962=23, srv1612514869=34, srv776765601=91, srv910446437=96, srv1168153621=12, srv297035347=67, srv1169633589=13, srv306988742=68, srv1857140602=46, srv2064892389=55, srv454132294=72, srv198441611=50, srv709485678=89, srv1370090283=25, srv1434196804=26, srv237759486=63, srv292841913=66, srv207842814=57, srv1293597841=22, srv34068987=71, srv53580214=84, srv90919390=95, srv1113383162=8, srv471803156=76, srv1174797896=14, srv1536437830=29, srv1162485188=11, srv459572118=73, srv1633218359=35, srv205077331=54, srv473777089=78, srv2029827628=53, srv478391326=79, srv1751020316=42, srv1205234383=16, srv2072798377=56, srv1240532509=17, srv12919135=21, srv737868675=90, srv996669060=99, srv2018549464=51, srv533961744=83, srv32055905=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-17 07:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-17 07:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-17 07:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-17 07:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-17 07:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-17 07:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-17 07:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-17 07:53:39,066 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-17 07:53:39,067 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-17 07:53:39,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-17 07:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-17 07:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-17 07:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-17 07:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-17 07:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-17 07:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-17 07:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-17 07:53:39,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-17 07:53:39,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-17 07:53:39,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-17 07:53:39,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-17 07:53:39,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-17 07:53:39,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-17 07:53:39,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-17 07:53:39,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-17 07:53:39,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-17 07:53:39,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-17 07:53:39,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-17 07:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-17 07:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-17 07:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-17 07:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-17 07:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-17 07:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-17 07:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-17 07:53:39,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-17 07:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-17 07:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-17 07:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-17 07:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-17 07:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-17 07:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-17 07:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-17 07:53:39,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-17 07:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-17 07:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-17 07:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-17 07:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-17 07:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-17 07:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-17 07:53:39,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-17 07:53:39,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-17 07:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-17 07:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-17 07:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-17 07:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-17 07:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-17 07:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-17 07:53:39,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-17 07:53:39,361 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:39,361 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-17 07:53:39,361 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-17 07:53:39,799 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 693 ms to try 5000 different iterations. Found a solution that moves 4670 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1776334774503691. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0401872410217505, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06005609050491054, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5188888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0401872410217505, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:39,814 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1434196804:82 , srv198441611:82 , srv1060178775:83 , srv1077663760:83 , srv1151582595:83 , srv1509622807:83 , srv1693468680:83 , srv170372505:83 , srv1751020316:83 , srv2029827628:83 , srv210341128:83 , srv235614059:83 , srv525158325:83 , srv533961744:83 , srv90919390:83 , srv1026735288:84 , srv1168153621:84 , srv1201998408:84 , srv1205234383:84 , srv1266315448:84 , srv1293597841:84 , srv1633218359:84 , srv1642396739:84 , srv1716776192:84 , srv1835408694:84 , srv2025639013:84 , srv205077331:84 , srv245008912:84 , srv471803156:84 , srv737868675:84 , srv807262758:84 , srv948799373:84 , srv1323375962:85 , srv1443409733:85 , srv1576001473:85 , srv1732743369:85 , srv1799970147:85 , srv1825561643:85 , srv292841913:85 , srv297035347:85 , srv340321010:85 , srv459890251:85 , srv472559042:85 , srv473777089:85 , srv520338735:85 , srv663023662:85 , srv158418878:86 , srv1861509348:86 , srv190125593:86 , srv2139195670:86 , srv217146624:86 , srv32055905:86 , srv498808830:86 , srv852733196:86 , srv996669060:86 , srv1090510460:87 , srv1102614534:87 , srv1127153572:87 , srv1240532509:87 , srv1271463110:87 , srv1536437830:87 , srv207842814:87 , srv870211856:87 , srv910446437:87 , srv1162485188:88 , srv1174797896:88 , srv1612514869:88 , srv1682717290:88 , srv1857140602:88 , srv34068987:88 , srv53580214:88 , srv615764351:88 , srv619344537:88 , srv709485678:88 , srv1031610187:89 , srv1052160628:89 , srv1113383162:89 , srv12919135:89 , srv1335021433:89 , srv1561009810:89 , srv1611393882:89 , srv2072798377:89 , srv237759486:89 , srv306988742:89 , srv478391326:89 , srv776765601:89 , srv110274250:90 , srv1169633589:90 , srv1275377394:90 , srv1370090283:90 , srv219780610:90 , srv1889565465:91 , srv2018549464:91 , srv2064892389:91 , srv569147418:91 , srv941440069:91 , srv278330364:93 , srv454132294:94 , srv470338834:95 , srv459572118:447 } 2023-04-17 07:53:39,820 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-17 07:53:39,821 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv569147418=85, srv1576001473=31, srv235614059=62, srv1031610187=1, srv1275377394=20, srv948799373=98, srv190125593=49, srv470338834=75, srv110274250=7, srv472559042=77, srv278330364=65, srv852733196=93, srv941440069=97, srv1102614534=6, srv340321010=70, srv520338735=81, srv870211856=94, srv1682717290=37, srv1201998408=15, srv1127153572=9, srv1052160628=2, srv1889565465=48, srv525158325=82, srv1799970147=43, srv1509622807=28, srv1732743369=41, srv1861509348=47, srv1271463110=19, srv1835408694=45, srv1026735288=0, srv1077663760=4, srv158418878=32, srv1090510460=5, srv1611393882=33, srv245008912=64, srv170372505=39, srv1335021433=24, srv210341128=58, srv807262758=92, srv1693468680=38, srv1151582595=10, srv498808830=80, srv615764351=86, srv217146624=60, srv1825561643=44, srv1266315448=18, srv2025639013=52, srv459890251=74, srv663023662=88, srv2139195670=59, srv219780610=61, srv1060178775=3, srv1443409733=27, srv1642396739=36, srv1716776192=40, srv1561009810=30, srv619344537=87, srv1323375962=23, srv1612514869=34, srv776765601=91, srv910446437=96, srv1168153621=12, srv297035347=67, srv1169633589=13, srv306988742=68, srv1857140602=46, srv2064892389=55, srv454132294=72, srv198441611=50, srv709485678=89, srv1370090283=25, srv1434196804=26, srv237759486=63, srv292841913=66, srv207842814=57, srv1293597841=22, srv34068987=71, srv53580214=84, srv90919390=95, srv1113383162=8, srv471803156=76, srv1174797896=14, srv1536437830=29, srv1162485188=11, srv459572118=73, srv1633218359=35, srv205077331=54, srv473777089=78, srv2029827628=53, srv478391326=79, srv1751020316=42, srv1205234383=16, srv2072798377=56, srv1240532509=17, srv12919135=21, srv737868675=90, srv996669060=99, srv2018549464=51, srv533961744=83, srv32055905=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-17 07:53:39,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-17 07:53:39,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-17 07:53:39,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-17 07:53:39,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-17 07:53:39,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-17 07:53:39,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-17 07:53:39,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-17 07:53:39,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-17 07:53:39,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-17 07:53:39,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-17 07:53:39,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-17 07:53:39,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-17 07:53:39,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-17 07:53:39,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-17 07:53:39,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-17 07:53:39,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-17 07:53:39,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-17 07:53:39,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-17 07:53:39,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-17 07:53:39,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-17 07:53:39,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-17 07:53:39,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-17 07:53:39,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-17 07:53:39,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-17 07:53:39,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-17 07:53:39,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-17 07:53:39,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-17 07:53:39,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-17 07:53:39,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-17 07:53:39,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-17 07:53:39,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-17 07:53:39,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-17 07:53:39,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-17 07:53:39,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-17 07:53:39,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-17 07:53:39,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-17 07:53:39,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-17 07:53:39,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-17 07:53:39,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-17 07:53:39,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-17 07:53:39,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-17 07:53:39,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-17 07:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-17 07:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-17 07:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-17 07:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-17 07:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-17 07:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-17 07:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-17 07:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-17 07:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-17 07:53:39,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-17 07:53:39,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-17 07:53:39,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-17 07:53:39,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-17 07:53:39,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-17 07:53:39,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-17 07:53:39,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-17 07:53:39,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-17 07:53:39,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-17 07:53:39,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-17 07:53:39,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-17 07:53:39,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-17 07:53:39,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-17 07:53:39,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-17 07:53:39,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-17 07:53:39,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-17 07:53:39,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-17 07:53:39,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-17 07:53:39,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-17 07:53:39,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-17 07:53:39,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-17 07:53:39,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-17 07:53:39,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-17 07:53:39,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-17 07:53:39,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-17 07:53:39,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-17 07:53:40,007 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0401872410217505, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06005609050491054, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0401872410217505, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:40,007 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-17 07:53:40,007 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17760077741145713, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0401872410217505, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06005609050491054, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0401872410217505, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23425, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-17 07:53:40,209 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 325 ms to try 5000 different iterations. Found a solution that moves 2610 regions; Going from a computed imbalance of 0.17760077741145713 to a new imbalance of 0.01530734129916419. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429810291386181); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038861462190022865, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429810291386181); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:40,220 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv470338834:75 , srv454132294:77 , srv278330364:79 , srv569147418:79 , srv1169633589:81 , srv1889565465:81 , srv2018549464:81 , srv2064892389:81 , srv2072798377:81 , srv776765601:81 , srv1174797896:82 , srv1275377394:82 , srv1612514869:82 , srv941440069:82 , srv1031610187:83 , srv1090510460:83 , srv110274250:83 , srv1370090283:83 , srv1611393882:83 , srv190125593:83 , srv615764351:83 , srv709485678:83 , srv870211856:83 , srv910446437:83 , srv1052160628:84 , srv1113383162:84 , srv1162485188:84 , srv1240532509:84 , srv1271463110:84 , srv1682717290:84 , srv1857140602:84 , srv1861509348:84 , srv237759486:84 , srv34068987:84 , srv53580214:84 , srv12919135:85 , srv1536437830:85 , srv1561009810:85 , srv158418878:85 , srv1799970147:85 , srv297035347:85 , srv306988742:85 , srv32055905:85 , srv478391326:85 , srv520338735:85 , srv619344537:85 , srv852733196:85 , srv996669060:85 , srv1060178775:86 , srv1102614534:86 , srv1127153572:86 , srv1168153621:86 , srv1576001473:86 , srv207842814:86 , srv2139195670:86 , srv217146624:86 , srv219780610:86 , srv245008912:86 , srv472559042:86 , srv473777089:86 , srv498808830:86 , srv1077663760:87 , srv1201998408:87 , srv1205234383:87 , srv1443409733:87 , srv1633218359:87 , srv1732743369:87 , srv1751020316:87 , srv2025639013:87 , srv459890251:87 , srv533961744:87 , srv663023662:87 , srv807262758:87 , srv1026735288:88 , srv1151582595:88 , srv1266315448:88 , srv1293597841:88 , srv1335021433:88 , srv1716776192:88 , srv1825561643:88 , srv210341128:88 , srv235614059:88 , srv292841913:88 , srv471803156:88 , srv737868675:88 , srv90919390:88 , srv1693468680:89 , srv198441611:89 , srv2029827628:89 , srv340321010:89 , srv1323375962:90 , srv1434196804:90 , srv1642396739:90 , srv170372505:90 , srv1835408694:90 , srv205077331:90 , srv948799373:90 , srv1509622807:91 , srv525158325:91 , srv459572118:543 } 2023-04-17 07:53:40,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-17 07:53:40,227 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv569147418=85, srv1576001473=31, srv235614059=62, srv1031610187=1, srv1275377394=20, srv948799373=98, srv190125593=49, srv470338834=75, srv110274250=7, srv472559042=77, srv278330364=65, srv852733196=93, srv941440069=97, srv1102614534=6, srv340321010=70, srv520338735=81, srv870211856=94, srv1682717290=37, srv1201998408=15, srv1127153572=9, srv1052160628=2, srv1889565465=48, srv525158325=82, srv1799970147=43, srv1509622807=28, srv1732743369=41, srv1861509348=47, srv1271463110=19, srv1835408694=45, srv1026735288=0, srv1077663760=4, srv158418878=32, srv1090510460=5, srv1611393882=33, srv245008912=64, srv170372505=39, srv1335021433=24, srv210341128=58, srv807262758=92, srv1693468680=38, srv1151582595=10, srv498808830=80, srv615764351=86, srv217146624=60, srv1825561643=44, srv1266315448=18, srv2025639013=52, srv459890251=74, srv663023662=88, srv2139195670=59, srv219780610=61, srv1060178775=3, srv1443409733=27, srv1642396739=36, srv1716776192=40, srv1561009810=30, srv619344537=87, srv1323375962=23, srv1612514869=34, srv776765601=91, srv910446437=96, srv1168153621=12, srv297035347=67, srv1169633589=13, srv306988742=68, srv1857140602=46, srv2064892389=55, srv454132294=72, srv198441611=50, srv709485678=89, srv1370090283=25, srv1434196804=26, srv237759486=63, srv292841913=66, srv207842814=57, srv1293597841=22, srv34068987=71, srv53580214=84, srv90919390=95, srv1113383162=8, srv471803156=76, srv1174797896=14, srv1536437830=29, srv1162485188=11, srv459572118=73, srv1633218359=35, srv205077331=54, srv473777089=78, srv2029827628=53, srv478391326=79, srv1751020316=42, srv1205234383=16, srv2072798377=56, srv1240532509=17, srv12919135=21, srv737868675=90, srv996669060=99, srv2018549464=51, srv533961744=83, srv32055905=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-17 07:53:40,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-17 07:53:40,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-17 07:53:40,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-17 07:53:40,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-17 07:53:40,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-17 07:53:40,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-17 07:53:40,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-17 07:53:40,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-17 07:53:40,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-17 07:53:40,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-17 07:53:40,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-17 07:53:40,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-17 07:53:40,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-17 07:53:40,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-17 07:53:40,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-17 07:53:40,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-17 07:53:40,353 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429810291386181); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038861462190022865, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429810291386181); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:40,353 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-17 07:53:40,353 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01528906568855173, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02429810291386181); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038861462190022865, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02429810291386181); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012333333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04325, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-17 07:53:40,521 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 250 ms to try 5000 different iterations. Found a solution that moves 975 regions; Going from a computed imbalance of 0.01528906568855173 to a new imbalance of 7.218300822664572E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008809975187300359); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011080766645411142); MoveCostFunction : (multiplier=7.0, imbalance=0.10833333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008809975187300359); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:40,523 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1323375962:81 , srv1335021433:81 , srv219780610:81 , srv1102614534:83 , srv1509622807:83 , srv1642396739:83 , srv292841913:83 , srv525158325:83 , srv941440069:83 , srv1052160628:84 , srv1127153572:84 , srv12919135:84 , srv1370090283:84 , srv1561009810:84 , srv1825561643:84 , srv1835408694:84 , srv205077331:84 , srv207842814:84 , srv306988742:84 , srv340321010:84 , srv459890251:84 , srv498808830:84 , srv948799373:84 , srv1031610187:85 , srv110274250:85 , srv1113383162:85 , srv1151582595:85 , srv1169633589:85 , srv1266315448:85 , srv1271463110:85 , srv1275377394:85 , srv1293597841:85 , srv1434196804:85 , srv1443409733:85 , srv1536437830:85 , srv1633218359:85 , srv1682717290:85 , srv170372505:85 , srv1732743369:85 , srv1857140602:85 , srv1889565465:85 , srv198441611:85 , srv2018549464:85 , srv2029827628:85 , srv235614059:85 , srv237759486:85 , srv278330364:85 , srv478391326:85 , srv53580214:85 , srv619344537:85 , srv663023662:85 , srv709485678:85 , srv1168153621:86 , srv1201998408:86 , srv1205234383:86 , srv1240532509:86 , srv1611393882:86 , srv1716776192:86 , srv1799970147:86 , srv1861509348:86 , srv2064892389:86 , srv210341128:86 , srv2139195670:86 , srv217146624:86 , srv245008912:86 , srv297035347:86 , srv32055905:86 , srv34068987:86 , srv472559042:86 , srv473777089:86 , srv533961744:86 , srv807262758:86 , srv852733196:86 , srv90919390:86 , srv910446437:86 , srv1026735288:87 , srv1060178775:87 , srv1077663760:87 , srv1090510460:87 , srv1162485188:87 , srv1174797896:87 , srv1576001473:87 , srv158418878:87 , srv1612514869:87 , srv1693468680:87 , srv2025639013:87 , srv2072798377:87 , srv454132294:87 , srv470338834:87 , srv471803156:87 , srv520338735:87 , srv737868675:87 , srv776765601:87 , srv870211856:87 , srv1751020316:88 , srv190125593:88 , srv569147418:88 , srv615764351:88 , srv996669060:89 , srv459572118:546 } 2023-04-17 07:53:40,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-17 07:53:40,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv569147418=85, srv1576001473=31, srv235614059=62, srv1031610187=1, srv1275377394=20, srv948799373=98, srv190125593=49, srv470338834=75, srv110274250=7, srv472559042=77, srv278330364=65, srv852733196=93, srv941440069=97, srv1102614534=6, srv340321010=70, srv520338735=81, srv870211856=94, srv1682717290=37, srv1201998408=15, srv1127153572=9, srv1052160628=2, srv1889565465=48, srv525158325=82, srv1799970147=43, srv1509622807=28, srv1732743369=41, srv1861509348=47, srv1271463110=19, srv1835408694=45, srv1026735288=0, srv1077663760=4, srv158418878=32, srv1090510460=5, srv1611393882=33, srv245008912=64, srv170372505=39, srv1335021433=24, srv210341128=58, srv807262758=92, srv1693468680=38, srv1151582595=10, srv498808830=80, srv615764351=86, srv217146624=60, srv1825561643=44, srv1266315448=18, srv2025639013=52, srv459890251=74, srv663023662=88, srv2139195670=59, srv219780610=61, srv1060178775=3, srv1443409733=27, srv1642396739=36, srv1716776192=40, srv1561009810=30, srv619344537=87, srv1323375962=23, srv1612514869=34, srv776765601=91, srv910446437=96, srv1168153621=12, srv297035347=67, srv1169633589=13, srv306988742=68, srv1857140602=46, srv2064892389=55, srv454132294=72, srv198441611=50, srv709485678=89, srv1370090283=25, srv1434196804=26, srv237759486=63, srv292841913=66, srv207842814=57, srv1293597841=22, srv34068987=71, srv53580214=84, srv90919390=95, srv1113383162=8, srv471803156=76, srv1174797896=14, srv1536437830=29, srv1162485188=11, srv459572118=73, srv1633218359=35, srv205077331=54, srv473777089=78, srv2029827628=53, srv478391326=79, srv1751020316=42, srv1205234383=16, srv2072798377=56, srv1240532509=17, srv12919135=21, srv737868675=90, srv996669060=99, srv2018549464=51, srv533961744=83, srv32055905=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-17 07:53:40,540 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-17 07:53:40,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-17 07:53:40,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-17 07:53:40,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-17 07:53:40,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-17 07:53:40,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-17 07:53:40,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-17 07:53:40,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-17 07:53:40,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-17 07:53:40,636 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008809975187300359); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011080766645411142); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008809975187300359); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:40,636 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-17 07:53:40,636 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.150029863480104E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008809975187300359); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011080766645411142); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008809975187300359); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-17 07:53:40,869 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 315 ms to try 5000 different iterations. Found a solution that moves 286 regions; Going from a computed imbalance of 7.150029863480104E-4 to a new imbalance of 2.8557264306114604E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.03177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:40,870 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv996669060:82 , srv1026735288:83 , srv1162485188:83 , srv1693468680:83 , srv237759486:83 , srv471803156:83 , srv478391326:83 , srv615764351:83 , srv737868675:83 , srv110274250:84 , srv1113383162:84 , srv12919135:84 , srv1335021433:84 , srv1536437830:84 , srv1561009810:84 , srv1576001473:84 , srv158418878:84 , srv1611393882:84 , srv170372505:84 , srv1751020316:84 , srv1835408694:84 , srv2025639013:84 , srv2029827628:84 , srv205077331:84 , srv2064892389:84 , srv217146624:84 , srv219780610:84 , srv278330364:84 , srv306988742:84 , srv340321010:84 , srv34068987:84 , srv454132294:84 , srv569147418:84 , srv619344537:84 , srv948799373:84 , srv1031610187:85 , srv1052160628:85 , srv1077663760:85 , srv1090510460:85 , srv1127153572:85 , srv1174797896:85 , srv1201998408:85 , srv1240532509:85 , srv1266315448:85 , srv1275377394:85 , srv1370090283:85 , srv1434196804:85 , srv1443409733:85 , srv1509622807:85 , srv1612514869:85 , srv1642396739:85 , srv1682717290:85 , srv1716776192:85 , srv1732743369:85 , srv1799970147:85 , srv1825561643:85 , srv1857140602:85 , srv190125593:85 , srv2018549464:85 , srv2072798377:85 , srv207842814:85 , srv210341128:85 , srv2139195670:85 , srv32055905:85 , srv470338834:85 , srv472559042:85 , srv473777089:85 , srv520338735:85 , srv525158325:85 , srv53580214:85 , srv663023662:85 , srv709485678:85 , srv776765601:85 , srv807262758:85 , srv852733196:85 , srv870211856:85 , srv1060178775:86 , srv1102614534:86 , srv1151582595:86 , srv1168153621:86 , srv1169633589:86 , srv1205234383:86 , srv1271463110:86 , srv1293597841:86 , srv1323375962:86 , srv1633218359:86 , srv1861509348:86 , srv1889565465:86 , srv198441611:86 , srv235614059:86 , srv245008912:86 , srv292841913:86 , srv297035347:86 , srv498808830:86 , srv533961744:86 , srv90919390:86 , srv910446437:86 , srv941440069:86 , srv459890251:87 , srv459572118:606 } 2023-04-17 07:53:40,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-17 07:53:40,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv569147418=85, srv1576001473=31, srv235614059=62, srv1031610187=1, srv1275377394=20, srv948799373=98, srv190125593=49, srv470338834=75, srv110274250=7, srv472559042=77, srv278330364=65, srv852733196=93, srv941440069=97, srv1102614534=6, srv340321010=70, srv520338735=81, srv870211856=94, srv1682717290=37, srv1201998408=15, srv1127153572=9, srv1052160628=2, srv1889565465=48, srv525158325=82, srv1799970147=43, srv1509622807=28, srv1732743369=41, srv1861509348=47, srv1271463110=19, srv1835408694=45, srv1026735288=0, srv1077663760=4, srv158418878=32, srv1090510460=5, srv1611393882=33, srv245008912=64, srv170372505=39, srv1335021433=24, srv210341128=58, srv807262758=92, srv1693468680=38, srv1151582595=10, srv498808830=80, srv615764351=86, srv217146624=60, srv1825561643=44, srv1266315448=18, srv2025639013=52, srv459890251=74, srv663023662=88, srv2139195670=59, srv219780610=61, srv1060178775=3, srv1443409733=27, srv1642396739=36, srv1716776192=40, srv1561009810=30, srv619344537=87, srv1323375962=23, srv1612514869=34, srv776765601=91, srv910446437=96, srv1168153621=12, srv297035347=67, srv1169633589=13, srv306988742=68, srv1857140602=46, srv2064892389=55, srv454132294=72, srv198441611=50, srv709485678=89, srv1370090283=25, srv1434196804=26, srv237759486=63, srv292841913=66, srv207842814=57, srv1293597841=22, srv34068987=71, srv53580214=84, srv90919390=95, srv1113383162=8, srv471803156=76, srv1174797896=14, srv1536437830=29, srv1162485188=11, srv459572118=73, srv1633218359=35, srv205077331=54, srv473777089=78, srv2029827628=53, srv478391326=79, srv1751020316=42, srv1205234383=16, srv2072798377=56, srv1240532509=17, srv12919135=21, srv737868675=90, srv996669060=99, srv2018549464=51, srv533961744=83, srv32055905=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-17 07:53:40,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-17 07:53:40,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-17 07:53:40,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-17 07:53:40,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-17 07:53:40,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-17 07:53:40,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-17 07:53:40,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-17 07:53:40,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-17 07:53:40,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-17 07:53:40,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-17 07:53:40,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-17 07:53:40,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-17 07:53:40,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-17 07:53:40,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-17 07:53:40,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-17 07:53:40,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-17 07:53:40,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-17 07:53:40,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-17 07:53:40,968 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:40,968 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-04-17 07:53:40,969 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.6554649503370158E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-04-17 07:53:41,122 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 225 ms to try 5000 different iterations. Found a solution that moves 29 regions; Going from a computed imbalance of 2.6554649503370158E-5 to a new imbalance of 7.590819840450981E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0032222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:41,123 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1205234383:83 , srv1293597841:83 , srv1716776192:83 , srv1889565465:83 , srv2139195670:83 , srv459890251:83 , srv90919390:83 , srv1026735288:84 , srv1031610187:84 , srv1052160628:84 , srv1060178775:84 , srv1077663760:84 , srv1090510460:84 , srv1102614534:84 , srv110274250:84 , srv1113383162:84 , srv1127153572:84 , srv1151582595:84 , srv1162485188:84 , srv1168153621:84 , srv1169633589:84 , srv1174797896:84 , srv1201998408:84 , srv1240532509:84 , srv1266315448:84 , srv1271463110:84 , srv1275377394:84 , srv12919135:84 , srv1323375962:84 , srv1335021433:84 , srv1370090283:84 , srv1434196804:84 , srv1443409733:84 , srv1509622807:84 , srv1561009810:84 , srv1576001473:84 , srv158418878:84 , srv1611393882:84 , srv1612514869:84 , srv1633218359:84 , srv1642396739:84 , srv1682717290:84 , srv1693468680:84 , srv170372505:84 , srv1732743369:84 , srv1751020316:84 , srv1825561643:84 , srv1835408694:84 , srv1857140602:84 , srv1861509348:84 , srv190125593:84 , srv198441611:84 , srv2018549464:84 , srv2025639013:84 , srv205077331:84 , srv2064892389:84 , srv2072798377:84 , srv207842814:84 , srv210341128:84 , srv217146624:84 , srv235614059:84 , srv245008912:84 , srv292841913:84 , srv297035347:84 , srv306988742:84 , srv32055905:84 , srv340321010:84 , srv34068987:84 , srv454132294:84 , srv470338834:84 , srv471803156:84 , srv472559042:84 , srv473777089:84 , srv478391326:84 , srv498808830:84 , srv520338735:84 , srv525158325:84 , srv533961744:84 , srv53580214:84 , srv569147418:84 , srv615764351:84 , srv619344537:84 , srv663023662:84 , srv737868675:84 , srv776765601:84 , srv807262758:84 , srv852733196:84 , srv870211856:84 , srv910446437:84 , srv941440069:84 , srv948799373:84 , srv996669060:84 , srv1536437830:85 , srv1799970147:85 , srv2029827628:85 , srv219780610:85 , srv237759486:85 , srv278330364:85 , srv709485678:85 , srv459572118:684 } 2023-04-17 07:53:41,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-17 07:53:41,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv569147418=85, srv1576001473=31, srv235614059=62, srv1031610187=1, srv1275377394=20, srv948799373=98, srv190125593=49, srv470338834=75, srv110274250=7, srv472559042=77, srv278330364=65, srv852733196=93, srv941440069=97, srv1102614534=6, srv340321010=70, srv520338735=81, srv870211856=94, srv1682717290=37, srv1201998408=15, srv1127153572=9, srv1052160628=2, srv1889565465=48, srv525158325=82, srv1799970147=43, srv1509622807=28, srv1732743369=41, srv1861509348=47, srv1271463110=19, srv1835408694=45, srv1026735288=0, srv1077663760=4, srv158418878=32, srv1090510460=5, srv1611393882=33, srv245008912=64, srv170372505=39, srv1335021433=24, srv210341128=58, srv807262758=92, srv1693468680=38, srv1151582595=10, srv498808830=80, srv615764351=86, srv217146624=60, srv1825561643=44, srv1266315448=18, srv2025639013=52, srv459890251=74, srv663023662=88, srv2139195670=59, srv219780610=61, srv1060178775=3, srv1443409733=27, srv1642396739=36, srv1716776192=40, srv1561009810=30, srv619344537=87, srv1323375962=23, srv1612514869=34, srv776765601=91, srv910446437=96, srv1168153621=12, srv297035347=67, srv1169633589=13, srv306988742=68, srv1857140602=46, srv2064892389=55, srv454132294=72, srv198441611=50, srv709485678=89, srv1370090283=25, srv1434196804=26, srv237759486=63, srv292841913=66, srv207842814=57, srv1293597841=22, srv34068987=71, srv53580214=84, srv90919390=95, srv1113383162=8, srv471803156=76, srv1174797896=14, srv1536437830=29, srv1162485188=11, srv459572118=73, srv1633218359=35, srv205077331=54, srv473777089=78, srv2029827628=53, srv478391326=79, srv1751020316=42, srv1205234383=16, srv2072798377=56, srv1240532509=17, srv12919135=21, srv737868675=90, srv996669060=99, srv2018549464=51, srv533961744=83, srv32055905=69} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-17 07:53:41,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-04-17 07:53:41,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-04-17 07:53:41,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-04-17 07:53:41,139 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-04-17 07:53:41,140 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-04-17 07:53:41,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-04-17 07:53:41,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-04-17 07:53:41,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-04-17 07:53:41,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-04-17 07:53:41,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-04-17 07:53:41,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-04-17 07:53:41,222 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.387757500312559E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0016412198797244363); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:41,223 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1205234383:83 , srv1293597841:83 , srv1716776192:83 , srv1889565465:83 , srv2139195670:83 , srv459890251:83 , srv90919390:83 , srv1026735288:84 , srv1031610187:84 , srv1052160628:84 , srv1060178775:84 , srv1077663760:84 , srv1090510460:84 , srv1102614534:84 , srv110274250:84 , srv1113383162:84 , srv1127153572:84 , srv1151582595:84 , srv1162485188:84 , srv1168153621:84 , srv1169633589:84 , srv1174797896:84 , srv1201998408:84 , srv1240532509:84 , srv1266315448:84 , srv1271463110:84 , srv1275377394:84 , srv12919135:84 , srv1323375962:84 , srv1335021433:84 , srv1370090283:84 , srv1434196804:84 , srv1443409733:84 , srv1509622807:84 , srv1561009810:84 , srv1576001473:84 , srv158418878:84 , srv1611393882:84 , srv1612514869:84 , srv1633218359:84 , srv1642396739:84 , srv1682717290:84 , srv1693468680:84 , srv170372505:84 , srv1732743369:84 , srv1751020316:84 , srv1825561643:84 , srv1835408694:84 , srv1857140602:84 , srv1861509348:84 , srv190125593:84 , srv198441611:84 , srv2018549464:84 , srv2025639013:84 , srv205077331:84 , srv2064892389:84 , srv2072798377:84 , srv207842814:84 , srv210341128:84 , srv217146624:84 , srv235614059:84 , srv245008912:84 , srv292841913:84 , srv297035347:84 , srv306988742:84 , srv32055905:84 , srv340321010:84 , srv34068987:84 , srv454132294:84 , srv470338834:84 , srv471803156:84 , srv472559042:84 , srv473777089:84 , srv478391326:84 , srv498808830:84 , srv520338735:84 , srv525158325:84 , srv533961744:84 , srv53580214:84 , srv569147418:84 , srv615764351:84 , srv619344537:84 , srv663023662:84 , srv737868675:84 , srv776765601:84 , srv807262758:84 , srv852733196:84 , srv870211856:84 , srv910446437:84 , srv941440069:84 , srv948799373:84 , srv996669060:84 , srv1536437830:85 , srv1799970147:85 , srv2029827628:85 , srv219780610:85 , srv237759486:85 , srv278330364:85 , srv709485678:85 , srv459572118:684 } 2023-04-17 07:53:41,252 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=316 (was 316), ProcessCount=173 (was 173), AvailableMemoryMB=8254 (was 8790) 2023-04-17 07:53:41,260 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=316, ProcessCount=173, AvailableMemoryMB=8254 2023-04-17 07:53:41,261 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-04-17 07:53:41,261 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-04-17 07:53:41,274 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1050721982:84 , srv1055183496:84 , srv1085123999:84 , srv1103754356:84 , srv1108054578:84 , srv1136659614:84 , srv1145565891:84 , srv1149982738:84 , srv1181035285:84 , srv1354301739:84 , srv1508751138:84 , srv1659342473:84 , srv1672806345:84 , srv1776162898:84 , srv1824005575:84 , srv1915165809:84 , srv261454970:84 , srv299216339:84 , srv380723276:84 , srv409796542:84 , srv434453253:84 , srv460036033:84 , srv482001290:84 , srv535666701:84 , srv598025604:84 , srv608284646:84 , srv632008162:84 , srv724981380:84 , srv767267456:84 , srv205492542:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-04-17 07:53:41,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-17 07:53:41,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181035285=8, srv1055183496=1, srv1659342473=11, srv1776162898=13, srv1145565891=6, srv598025604=25, srv1508751138=10, srv482001290=23, srv1108054578=4, srv205492542=16, srv1824005575=14, srv460036033=22, srv1354301739=9, srv1136659614=5, srv261454970=17, srv632008162=27, srv1085123999=2, srv1149982738=7, srv1103754356=3, srv1672806345=12, srv535666701=24, srv380723276=19, srv767267456=29, srv1050721982=0, srv608284646=26, srv1915165809=15, srv434453253=21, srv724981380=28, srv299216339=18, srv409796542=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-17 07:53:41,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-17 07:53:41,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-17 07:53:41,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-17 07:53:41,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-17 07:53:41,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-17 07:53:41,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-17 07:53:41,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-17 07:53:41,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-17 07:53:41,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-17 07:53:41,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-17 07:53:41,316 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1308770078184232, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:53:41,317 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912605432442388, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1308770078184232, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-04-17 07:55:41,294 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9824822 different iterations. Found a solution that moves 1846 regions; Going from a computed imbalance of 0.9912605432442388 to a new imbalance of 4.308656090753194E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6837037037037037, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:55:41,300 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1050721982:90 , srv1055183496:90 , srv1085123999:90 , srv1103754356:90 , srv1108054578:90 , srv1136659614:90 , srv1145565891:90 , srv1149982738:90 , srv1181035285:90 , srv1354301739:90 , srv1508751138:90 , srv1659342473:90 , srv1672806345:90 , srv1776162898:90 , srv1824005575:90 , srv1915165809:90 , srv205492542:90 , srv261454970:90 , srv299216339:90 , srv380723276:90 , srv409796542:90 , srv434453253:90 , srv460036033:90 , srv482001290:90 , srv535666701:90 , srv598025604:90 , srv608284646:90 , srv632008162:90 , srv724981380:90 , srv767267456:90 } 2023-04-17 07:55:41,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-04-17 07:55:41,302 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1181035285=8, srv1055183496=1, srv1659342473=11, srv1776162898=13, srv1145565891=6, srv598025604=25, srv1508751138=10, srv482001290=23, srv1108054578=4, srv205492542=16, srv1824005575=14, srv460036033=22, srv1354301739=9, srv1136659614=5, srv261454970=17, srv632008162=27, srv1085123999=2, srv1149982738=7, srv1103754356=3, srv1672806345=12, srv535666701=24, srv380723276=19, srv767267456=29, srv1050721982=0, srv608284646=26, srv1915165809=15, srv434453253=21, srv724981380=28, srv299216339=18, srv409796542=20} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-04-17 07:55:41,307 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-04-17 07:55:41,308 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-04-17 07:55:41,319 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-04-17 07:55:41,320 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1050721982:90 , srv1055183496:90 , srv1085123999:90 , srv1103754356:90 , srv1108054578:90 , srv1136659614:90 , srv1145565891:90 , srv1149982738:90 , srv1181035285:90 , srv1354301739:90 , srv1508751138:90 , srv1659342473:90 , srv1672806345:90 , srv1776162898:90 , srv1824005575:90 , srv1915165809:90 , srv205492542:90 , srv261454970:90 , srv299216339:90 , srv380723276:90 , srv409796542:90 , srv434453253:90 , srv460036033:90 , srv482001290:90 , srv535666701:90 , srv598025604:90 , srv608284646:90 , srv632008162:90 , srv724981380:90 , srv767267456:90 } 2023-04-17 07:55:41,330 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=166 (was 316), ProcessCount=168 (was 173), AvailableMemoryMB=8418 (was 8254) - AvailableMemoryMB LEAK? -