2023-03-26 23:56:04,783 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins 2023-03-26 23:56:05,145 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 23:56:05,275 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-26 23:56:05,362 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-26 23:56:05,372 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-26 23:56:05,391 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=441, ProcessCount=170, AvailableMemoryMB=9450 2023-03-26 23:56:05,394 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-26 23:56:05,395 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-26 23:56:05,564 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1029819848:84 , srv103633515:84 , srv1063687939:84 , srv1114951062:84 , srv1129598405:84 , srv1129940077:84 , srv1136497730:84 , srv1142196637:84 , srv11493732:84 , srv1188656362:84 , srv1188658396:84 , srv1221962339:84 , srv1252055430:84 , srv1284957953:84 , srv1285869800:84 , srv1314353267:84 , srv1314683193:84 , srv1322587707:84 , srv1332809676:84 , srv1334877849:84 , srv1387818344:84 , srv142900490:84 , srv1462063112:84 , srv1469441281:84 , srv1488021782:84 , srv1528055168:84 , srv1554669260:84 , srv1556540352:84 , srv1565563239:84 , srv1576553755:84 , srv1589528626:84 , srv1641595621:84 , srv1643789669:84 , srv1684704928:84 , srv1698895345:84 , srv1699459974:84 , srv1721506478:84 , srv1727642323:84 , srv1731050689:84 , srv1746492335:84 , srv1746872516:84 , srv1792589038:84 , srv1798857200:84 , srv1827128367:84 , srv1860304014:84 , srv1865799478:84 , srv1904088499:84 , srv1940882407:84 , srv1941155225:84 , srv1948889682:84 , srv1971506394:84 , srv1988803963:84 , srv2014991335:84 , srv2084071354:84 , srv2107897377:84 , srv2121437375:84 , srv2144651147:84 , srv244852309:84 , srv265346526:84 , srv27779793:84 , srv2923149:84 , srv322062264:84 , srv340043714:84 , srv36102291:84 , srv362783679:84 , srv364803077:84 , srv378527039:84 , srv386357860:84 , srv422913317:84 , srv428174220:84 , srv477655458:84 , srv523253324:84 , srv54506157:84 , srv553385763:84 , srv564266418:84 , srv591844348:84 , srv592709363:84 , srv613665936:84 , srv63487246:84 , srv641435057:84 , srv684711910:84 , srv685497201:84 , srv713817160:84 , srv7162090:84 , srv72524773:84 , srv728232652:84 , srv748031309:84 , srv813302350:84 , srv862013239:84 , srv865193259:84 , srv891272449:84 , srv898435715:84 , srv911544909:84 , srv911839344:84 , srv915542388:84 , srv924872609:84 , srv94660692:84 , srv948374757:84 , srv970829622:84 , srv700315327:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90] 2023-03-26 23:56:05,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-26 23:56:05,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1314353267=15, srv1641595621=31, srv1322587707=17, srv813302350=88, srv1556540352=27, srv1589528626=30, srv591844348=75, srv1332809676=18, srv1643789669=32, srv1971506394=50, srv362783679=64, srv1129940077=5, srv94660692=97, srv1063687939=2, srv2107897377=54, srv641435057=79, srv1554669260=26, srv1948889682=49, srv428174220=69, srv1684704928=33, srv1904088499=46, srv1565563239=28, srv1488021782=24, srv1792589038=41, srv728232652=86, srv1727642323=37, srv1865799478=45, srv63487246=78, srv1136497730=6, srv564266418=74, srv1029819848=0, srv891272449=91, srv553385763=73, srv1387818344=20, srv322062264=61, srv1284957953=13, srv1988803963=51, srv1698895345=34, srv378527039=66, srv865193259=90, srv1221962339=11, srv1129598405=4, srv592709363=76, srv1699459974=35, srv27779793=59, srv2923149=60, srv54506157=72, srv915542388=95, srv142900490=21, srv911839344=94, srv700315327=82, srv948374757=98, srv613665936=77, srv2014991335=52, srv924872609=96, srv103633515=1, srv1188656362=9, srv422913317=68, srv1576553755=29, srv2084071354=53, srv2144651147=56, srv684711910=80, srv1252055430=12, srv1746872516=40, srv386357860=67, srv2121437375=55, srv685497201=81, srv1285869800=14, srv1746492335=39, srv1314683193=16, srv1142196637=7, srv1731050689=38, srv1188658396=10, srv265346526=58, srv11493732=8, srv862013239=89, srv1860304014=44, srv340043714=62, srv1721506478=36, srv748031309=87, srv713817160=83, srv244852309=57, srv36102291=63, srv1827128367=43, srv523253324=71, srv72524773=85, srv477655458=70, srv1528055168=25, srv1114951062=3, srv1462063112=22, srv898435715=92, srv970829622=99, srv911544909=93, srv1798857200=42, srv1940882407=47, srv1941155225=48, srv7162090=84, srv1334877849=19, srv1469441281=23, srv364803077=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-26 23:56:05,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-26 23:56:05,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-26 23:56:05,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-26 23:56:05,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-26 23:56:05,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-26 23:56:05,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-26 23:56:05,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-26 23:56:05,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-26 23:56:05,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-26 23:56:05,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-26 23:56:05,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-26 23:56:05,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-26 23:56:05,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-26 23:56:05,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-26 23:56:05,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-26 23:56:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-26 23:56:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-26 23:56:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-26 23:56:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-26 23:56:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-26 23:56:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-26 23:56:05,723 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-26 23:56:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-26 23:56:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-26 23:56:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-26 23:56:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-26 23:56:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-26 23:56:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-26 23:56:05,724 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-26 23:56:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-26 23:56:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-26 23:56:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-26 23:56:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-26 23:56:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-26 23:56:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-26 23:56:05,725 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-26 23:56:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-26 23:56:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-26 23:56:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-26 23:56:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-26 23:56:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-26 23:56:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-26 23:56:05,726 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-26 23:56:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-26 23:56:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-26 23:56:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-26 23:56:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-26 23:56:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-26 23:56:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-26 23:56:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-26 23:56:05,727 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-26 23:56:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-26 23:56:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-26 23:56:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-26 23:56:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-26 23:56:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-26 23:56:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-26 23:56:05,728 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-26 23:56:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-26 23:56:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-26 23:56:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-26 23:56:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-26 23:56:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-26 23:56:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-26 23:56:05,729 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-26 23:56:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-26 23:56:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-26 23:56:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-26 23:56:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-26 23:56:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-26 23:56:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-26 23:56:05,730 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-26 23:56:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-26 23:56:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-26 23:56:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-26 23:56:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-26 23:56:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-26 23:56:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-26 23:56:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-26 23:56:05,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-26 23:56:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-26 23:56:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-26 23:56:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-26 23:56:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-26 23:56:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-26 23:56:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-26 23:56:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-26 23:56:05,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-26 23:56:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-26 23:56:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-26 23:56:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-26 23:56:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-26 23:56:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-26 23:56:05,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-26 23:56:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-26 23:56:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-26 23:56:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-26 23:56:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-26 23:56:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-26 23:56:05,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-26 23:56:05,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-26 23:56:05,985 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:05,986 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-26 23:56:05,986 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-26 23:56:06,350 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 582 ms to try 5000 different iterations. Found a solution that moves 4686 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18115887382973273. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914569688021195, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05932656110861463, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5206666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914569688021195, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24766666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:06,368 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv728232652:81 , srv1221962339:83 , srv1284957953:83 , srv1684704928:83 , srv1948889682:83 , srv27779793:83 , srv36102291:83 , srv7162090:83 , srv72524773:83 , srv1029819848:84 , srv103633515:84 , srv1142196637:84 , srv1699459974:84 , srv1727642323:84 , srv1860304014:84 , srv1865799478:84 , srv1941155225:84 , srv378527039:84 , srv94660692:84 , srv1063687939:85 , srv11493732:85 , srv1314353267:85 , srv1469441281:85 , srv1589528626:85 , srv1746492335:85 , srv1746872516:85 , srv1792589038:85 , srv1798857200:85 , srv1827128367:85 , srv1940882407:85 , srv1988803963:85 , srv2107897377:85 , srv362783679:85 , srv364803077:85 , srv422913317:85 , srv477655458:85 , srv523253324:85 , srv54506157:85 , srv862013239:85 , srv915542388:85 , srv1188658396:86 , srv1387818344:86 , srv1641595621:86 , srv1721506478:86 , srv1904088499:86 , srv1971506394:86 , srv244852309:86 , srv322062264:86 , srv684711910:86 , srv748031309:86 , srv970829622:86 , srv1114951062:87 , srv1188656362:87 , srv1322587707:87 , srv142900490:87 , srv1488021782:87 , srv1565563239:87 , srv1576553755:87 , srv1643789669:87 , srv2084071354:87 , srv265346526:87 , srv386357860:87 , srv428174220:87 , srv564266418:87 , srv591844348:87 , srv613665936:87 , srv641435057:87 , srv685497201:87 , srv713817160:87 , srv891272449:87 , srv911544909:87 , srv924872609:87 , srv1252055430:88 , srv1556540352:88 , srv2121437375:88 , srv592709363:88 , srv898435715:88 , srv911839344:88 , srv948374757:88 , srv1129598405:89 , srv1314683193:89 , srv1528055168:89 , srv1554669260:89 , srv1731050689:89 , srv2014991335:89 , srv813302350:89 , srv865193259:89 , srv1334877849:90 , srv2923149:90 , srv340043714:90 , srv553385763:90 , srv1285869800:91 , srv1332809676:91 , srv1462063112:91 , srv1698895345:91 , srv2144651147:91 , srv63487246:91 , srv1129940077:92 , srv1136497730:93 , srv700315327:438 } 2023-03-26 23:56:06,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-26 23:56:06,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1314353267=15, srv1641595621=31, srv1322587707=17, srv813302350=88, srv1556540352=27, srv1589528626=30, srv591844348=75, srv1332809676=18, srv1643789669=32, srv1971506394=50, srv362783679=64, srv1129940077=5, srv94660692=97, srv1063687939=2, srv2107897377=54, srv641435057=79, srv1554669260=26, srv1948889682=49, srv428174220=69, srv1684704928=33, srv1904088499=46, srv1565563239=28, srv1488021782=24, srv1792589038=41, srv728232652=86, srv1727642323=37, srv1865799478=45, srv63487246=78, srv1136497730=6, srv564266418=74, srv1029819848=0, srv891272449=91, srv553385763=73, srv1387818344=20, srv322062264=61, srv1284957953=13, srv1988803963=51, srv1698895345=34, srv378527039=66, srv865193259=90, srv1221962339=11, srv1129598405=4, srv592709363=76, srv1699459974=35, srv27779793=59, srv2923149=60, srv54506157=72, srv915542388=95, srv142900490=21, srv911839344=94, srv700315327=82, srv948374757=98, srv613665936=77, srv2014991335=52, srv924872609=96, srv103633515=1, srv1188656362=9, srv422913317=68, srv1576553755=29, srv2084071354=53, srv2144651147=56, srv684711910=80, srv1252055430=12, srv1746872516=40, srv386357860=67, srv2121437375=55, srv685497201=81, srv1285869800=14, srv1746492335=39, srv1314683193=16, srv1142196637=7, srv1731050689=38, srv1188658396=10, srv265346526=58, srv11493732=8, srv862013239=89, srv1860304014=44, srv340043714=62, srv1721506478=36, srv748031309=87, srv713817160=83, srv244852309=57, srv36102291=63, srv1827128367=43, srv523253324=71, srv72524773=85, srv477655458=70, srv1528055168=25, srv1114951062=3, srv1462063112=22, srv898435715=92, srv970829622=99, srv911544909=93, srv1798857200=42, srv1940882407=47, srv1941155225=48, srv7162090=84, srv1334877849=19, srv1469441281=23, srv364803077=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-26 23:56:06,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-26 23:56:06,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-26 23:56:06,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-26 23:56:06,403 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-26 23:56:06,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-26 23:56:06,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-26 23:56:06,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-26 23:56:06,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-26 23:56:06,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-26 23:56:06,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-26 23:56:06,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-26 23:56:06,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-26 23:56:06,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-26 23:56:06,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-26 23:56:06,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-26 23:56:06,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-26 23:56:06,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-26 23:56:06,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-26 23:56:06,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-26 23:56:06,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-26 23:56:06,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-26 23:56:06,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-26 23:56:06,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-26 23:56:06,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-26 23:56:06,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-26 23:56:06,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-26 23:56:06,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-26 23:56:06,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-26 23:56:06,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-26 23:56:06,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-26 23:56:06,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-26 23:56:06,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-26 23:56:06,407 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-26 23:56:06,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-26 23:56:06,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-26 23:56:06,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-26 23:56:06,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-26 23:56:06,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-26 23:56:06,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-26 23:56:06,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-26 23:56:06,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-26 23:56:06,409 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-26 23:56:06,410 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-26 23:56:06,411 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-26 23:56:06,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-26 23:56:06,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-26 23:56:06,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-26 23:56:06,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-26 23:56:06,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-26 23:56:06,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-26 23:56:06,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-26 23:56:06,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-26 23:56:06,598 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914569688021195, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05932656110861463, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914569688021195, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24766666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:06,598 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-26 23:56:06,599 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18112606175642626, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03914569688021195, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05932656110861463, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03914569688021195, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17591666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24766666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-26 23:56:06,962 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 521 ms to try 5000 different iterations. Found a solution that moves 2645 regions; Going from a computed imbalance of 0.18112606175642626 to a new imbalance of 0.015692141183718568. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02395385679319171); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03632686131465511, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29388888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02395385679319171); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.046, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:06,975 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1136497730:79 , srv1698895345:79 , srv1129940077:80 , srv1334877849:80 , srv2144651147:80 , srv340043714:80 , srv63487246:80 , srv1129598405:81 , srv1285869800:81 , srv1554669260:81 , srv1731050689:81 , srv553385763:81 , srv1314683193:82 , srv1332809676:82 , srv1462063112:82 , srv1528055168:82 , srv813302350:82 , srv1322587707:83 , srv1576553755:83 , srv2923149:83 , srv592709363:83 , srv865193259:83 , srv911544909:83 , srv924872609:83 , srv948374757:83 , srv1114951062:84 , srv1252055430:84 , srv142900490:84 , srv1565563239:84 , srv2014991335:84 , srv564266418:84 , srv641435057:84 , srv898435715:84 , srv1188658396:85 , srv1488021782:85 , srv1556540352:85 , srv1721506478:85 , srv1746492335:85 , srv1792589038:85 , srv1827128367:85 , srv2121437375:85 , srv244852309:85 , srv265346526:85 , srv428174220:85 , srv523253324:85 , srv685497201:85 , srv915542388:85 , srv94660692:85 , srv1063687939:86 , srv1188656362:86 , srv1641595621:86 , srv1746872516:86 , srv1798857200:86 , srv1940882407:86 , srv1971506394:86 , srv2107897377:86 , srv322062264:86 , srv364803077:86 , srv613665936:86 , srv748031309:86 , srv891272449:86 , srv911839344:86 , srv1142196637:87 , srv11493732:87 , srv1314353267:87 , srv1589528626:87 , srv1684704928:87 , srv1860304014:87 , srv1904088499:87 , srv2084071354:87 , srv36102291:87 , srv386357860:87 , srv54506157:87 , srv591844348:87 , srv684711910:87 , srv713817160:87 , srv970829622:87 , srv1029819848:88 , srv1387818344:88 , srv1469441281:88 , srv1865799478:88 , srv27779793:88 , srv362783679:88 , srv378527039:88 , srv422913317:88 , srv477655458:88 , srv862013239:88 , srv1284957953:89 , srv1643789669:89 , srv1727642323:89 , srv1941155225:89 , srv1988803963:89 , srv7162090:89 , srv103633515:90 , srv1948889682:90 , srv1221962339:91 , srv1699459974:91 , srv72524773:91 , srv728232652:91 , srv700315327:549 } 2023-03-26 23:56:06,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-26 23:56:06,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1314353267=15, srv1641595621=31, srv1322587707=17, srv813302350=88, srv1556540352=27, srv1589528626=30, srv591844348=75, srv1332809676=18, srv1643789669=32, srv1971506394=50, srv362783679=64, srv1129940077=5, srv94660692=97, srv1063687939=2, srv2107897377=54, srv641435057=79, srv1554669260=26, srv1948889682=49, srv428174220=69, srv1684704928=33, srv1904088499=46, srv1565563239=28, srv1488021782=24, srv1792589038=41, srv728232652=86, srv1727642323=37, srv1865799478=45, srv63487246=78, srv1136497730=6, srv564266418=74, srv1029819848=0, srv891272449=91, srv553385763=73, srv1387818344=20, srv322062264=61, srv1284957953=13, srv1988803963=51, srv1698895345=34, srv378527039=66, srv865193259=90, srv1221962339=11, srv1129598405=4, srv592709363=76, srv1699459974=35, srv27779793=59, srv2923149=60, srv54506157=72, srv915542388=95, srv142900490=21, srv911839344=94, srv700315327=82, srv948374757=98, srv613665936=77, srv2014991335=52, srv924872609=96, srv103633515=1, srv1188656362=9, srv422913317=68, srv1576553755=29, srv2084071354=53, srv2144651147=56, srv684711910=80, srv1252055430=12, srv1746872516=40, srv386357860=67, srv2121437375=55, srv685497201=81, srv1285869800=14, srv1746492335=39, srv1314683193=16, srv1142196637=7, srv1731050689=38, srv1188658396=10, srv265346526=58, srv11493732=8, srv862013239=89, srv1860304014=44, srv340043714=62, srv1721506478=36, srv748031309=87, srv713817160=83, srv244852309=57, srv36102291=63, srv1827128367=43, srv523253324=71, srv72524773=85, srv477655458=70, srv1528055168=25, srv1114951062=3, srv1462063112=22, srv898435715=92, srv970829622=99, srv911544909=93, srv1798857200=42, srv1940882407=47, srv1941155225=48, srv7162090=84, srv1334877849=19, srv1469441281=23, srv364803077=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-26 23:56:07,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-26 23:56:07,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-26 23:56:07,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-26 23:56:07,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-26 23:56:07,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-26 23:56:07,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-26 23:56:07,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-26 23:56:07,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-26 23:56:07,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-26 23:56:07,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-26 23:56:07,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-26 23:56:07,010 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-26 23:56:07,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-26 23:56:07,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-26 23:56:07,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-26 23:56:07,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-26 23:56:07,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-26 23:56:07,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-26 23:56:07,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-26 23:56:07,011 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-26 23:56:07,177 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02395385679319171); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03632686131465511, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02395385679319171); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.046, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:07,177 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-26 23:56:07,178 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01567362049786801, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02395385679319171); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03632686131465511, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02395385679319171); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0125); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.046, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-26 23:56:07,523 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 498 ms to try 5000 different iterations. Found a solution that moves 976 regions; Going from a computed imbalance of 0.01567362049786801 to a new imbalance of 0.0013276706759813744. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010478036578004071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01221762999127965); MoveCostFunction : (multiplier=7.0, imbalance=0.10844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.010478036578004071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:07,526 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1699459974:81 , srv1221962339:82 , srv1643789669:82 , srv713817160:82 , srv72524773:82 , srv1387818344:83 , srv2084071354:83 , srv2121437375:83 , srv591844348:83 , srv891272449:83 , srv911839344:83 , srv103633515:84 , srv1188656362:84 , srv1332809676:84 , srv1462063112:84 , srv1641595621:84 , srv1727642323:84 , srv1904088499:84 , srv1941155225:84 , srv1948889682:84 , srv1988803963:84 , srv265346526:84 , srv2923149:84 , srv322062264:84 , srv362783679:84 , srv386357860:84 , srv477655458:84 , srv613665936:84 , srv641435057:84 , srv685497201:84 , srv748031309:84 , srv862013239:84 , srv970829622:84 , srv1029819848:85 , srv1129940077:85 , srv1142196637:85 , srv11493732:85 , srv1188658396:85 , srv1285869800:85 , srv1314353267:85 , srv1469441281:85 , srv1488021782:85 , srv1721506478:85 , srv1746872516:85 , srv1971506394:85 , srv378527039:85 , srv422913317:85 , srv54506157:85 , srv684711910:85 , srv7162090:85 , srv728232652:85 , srv1063687939:86 , srv1114951062:86 , srv1136497730:86 , srv1284957953:86 , srv1314683193:86 , srv1334877849:86 , srv142900490:86 , srv1528055168:86 , srv1556540352:86 , srv1576553755:86 , srv1589528626:86 , srv1792589038:86 , srv1798857200:86 , srv1827128367:86 , srv1865799478:86 , srv2014991335:86 , srv2107897377:86 , srv2144651147:86 , srv244852309:86 , srv27779793:86 , srv36102291:86 , srv364803077:86 , srv428174220:86 , srv523253324:86 , srv553385763:86 , srv592709363:86 , srv63487246:86 , srv813302350:86 , srv898435715:86 , srv915542388:86 , srv948374757:86 , srv1252055430:87 , srv1554669260:87 , srv1565563239:87 , srv1698895345:87 , srv1731050689:87 , srv1746492335:87 , srv1860304014:87 , srv340043714:87 , srv564266418:87 , srv865193259:87 , srv911544909:87 , srv924872609:87 , srv1322587707:88 , srv1684704928:88 , srv1940882407:88 , srv94660692:88 , srv1129598405:89 , srv700315327:564 } 2023-03-26 23:56:07,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-26 23:56:07,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1314353267=15, srv1641595621=31, srv1322587707=17, srv813302350=88, srv1556540352=27, srv1589528626=30, srv591844348=75, srv1332809676=18, srv1643789669=32, srv1971506394=50, srv362783679=64, srv1129940077=5, srv94660692=97, srv1063687939=2, srv2107897377=54, srv641435057=79, srv1554669260=26, srv1948889682=49, srv428174220=69, srv1684704928=33, srv1904088499=46, srv1565563239=28, srv1488021782=24, srv1792589038=41, srv728232652=86, srv1727642323=37, srv1865799478=45, srv63487246=78, srv1136497730=6, srv564266418=74, srv1029819848=0, srv891272449=91, srv553385763=73, srv1387818344=20, srv322062264=61, srv1284957953=13, srv1988803963=51, srv1698895345=34, srv378527039=66, srv865193259=90, srv1221962339=11, srv1129598405=4, srv592709363=76, srv1699459974=35, srv27779793=59, srv2923149=60, srv54506157=72, srv915542388=95, srv142900490=21, srv911839344=94, srv700315327=82, srv948374757=98, srv613665936=77, srv2014991335=52, srv924872609=96, srv103633515=1, srv1188656362=9, srv422913317=68, srv1576553755=29, srv2084071354=53, srv2144651147=56, srv684711910=80, srv1252055430=12, srv1746872516=40, srv386357860=67, srv2121437375=55, srv685497201=81, srv1285869800=14, srv1746492335=39, srv1314683193=16, srv1142196637=7, srv1731050689=38, srv1188658396=10, srv265346526=58, srv11493732=8, srv862013239=89, srv1860304014=44, srv340043714=62, srv1721506478=36, srv748031309=87, srv713817160=83, srv244852309=57, srv36102291=63, srv1827128367=43, srv523253324=71, srv72524773=85, srv477655458=70, srv1528055168=25, srv1114951062=3, srv1462063112=22, srv898435715=92, srv970829622=99, srv911544909=93, srv1798857200=42, srv1940882407=47, srv1941155225=48, srv7162090=84, srv1334877849=19, srv1469441281=23, srv364803077=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-26 23:56:07,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-26 23:56:07,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-26 23:56:07,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-26 23:56:07,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-26 23:56:07,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-26 23:56:07,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-26 23:56:07,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-26 23:56:07,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-26 23:56:07,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-26 23:56:07,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-26 23:56:07,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-26 23:56:07,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-26 23:56:07,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-26 23:56:07,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-26 23:56:07,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-26 23:56:07,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-26 23:56:07,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-26 23:56:07,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-26 23:56:07,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-26 23:56:07,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-26 23:56:07,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-26 23:56:07,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-26 23:56:07,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-26 23:56:07,721 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010478036578004071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01221762999127965); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.010478036578004071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:07,721 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-26 23:56:07,722 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0013208365779132675, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010478036578004071); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01221762999127965); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.010478036578004071); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333334E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005166666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-26 23:56:08,012 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 424 ms to try 5000 different iterations. Found a solution that moves 311 regions; Going from a computed imbalance of 0.0013208365779132675 to a new imbalance of 2.811830715063856E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.034555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:08,013 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1589528626:83 , srv1988803963:83 , srv2014991335:83 , srv386357860:83 , srv865193259:83 , srv103633515:84 , srv1129598405:84 , srv1136497730:84 , srv1252055430:84 , srv1284957953:84 , srv1322587707:84 , srv1554669260:84 , srv1556540352:84 , srv1565563239:84 , srv1643789669:84 , srv1684704928:84 , srv1798857200:84 , srv1865799478:84 , srv1940882407:84 , srv422913317:84 , srv428174220:84 , srv477655458:84 , srv564266418:84 , srv684711910:84 , srv728232652:84 , srv898435715:84 , srv1029819848:85 , srv1063687939:85 , srv1114951062:85 , srv1129940077:85 , srv11493732:85 , srv1188656362:85 , srv1188658396:85 , srv1285869800:85 , srv1314353267:85 , srv1314683193:85 , srv1332809676:85 , srv1334877849:85 , srv1387818344:85 , srv142900490:85 , srv1462063112:85 , srv1469441281:85 , srv1488021782:85 , srv1528055168:85 , srv1698895345:85 , srv1727642323:85 , srv1731050689:85 , srv1746492335:85 , srv1860304014:85 , srv1904088499:85 , srv1941155225:85 , srv1948889682:85 , srv1971506394:85 , srv2084071354:85 , srv2107897377:85 , srv2144651147:85 , srv244852309:85 , srv27779793:85 , srv2923149:85 , srv340043714:85 , srv362783679:85 , srv364803077:85 , srv378527039:85 , srv523253324:85 , srv54506157:85 , srv553385763:85 , srv591844348:85 , srv592709363:85 , srv613665936:85 , srv63487246:85 , srv7162090:85 , srv813302350:85 , srv862013239:85 , srv911544909:85 , srv911839344:85 , srv924872609:85 , srv94660692:85 , srv948374757:85 , srv970829622:85 , srv1142196637:86 , srv1221962339:86 , srv1699459974:86 , srv1721506478:86 , srv1746872516:86 , srv1792589038:86 , srv1827128367:86 , srv2121437375:86 , srv265346526:86 , srv322062264:86 , srv685497201:86 , srv713817160:86 , srv72524773:86 , srv748031309:86 , srv891272449:86 , srv915542388:86 , srv1576553755:87 , srv1641595621:87 , srv36102291:87 , srv641435057:87 , srv700315327:592 } 2023-03-26 23:56:08,016 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-26 23:56:08,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1314353267=15, srv1641595621=31, srv1322587707=17, srv813302350=88, srv1556540352=27, srv1589528626=30, srv591844348=75, srv1332809676=18, srv1643789669=32, srv1971506394=50, srv362783679=64, srv1129940077=5, srv94660692=97, srv1063687939=2, srv2107897377=54, srv641435057=79, srv1554669260=26, srv1948889682=49, srv428174220=69, srv1684704928=33, srv1904088499=46, srv1565563239=28, srv1488021782=24, srv1792589038=41, srv728232652=86, srv1727642323=37, srv1865799478=45, srv63487246=78, srv1136497730=6, srv564266418=74, srv1029819848=0, srv891272449=91, srv553385763=73, srv1387818344=20, srv322062264=61, srv1284957953=13, srv1988803963=51, srv1698895345=34, srv378527039=66, srv865193259=90, srv1221962339=11, srv1129598405=4, srv592709363=76, srv1699459974=35, srv27779793=59, srv2923149=60, srv54506157=72, srv915542388=95, srv142900490=21, srv911839344=94, srv700315327=82, srv948374757=98, srv613665936=77, srv2014991335=52, srv924872609=96, srv103633515=1, srv1188656362=9, srv422913317=68, srv1576553755=29, srv2084071354=53, srv2144651147=56, srv684711910=80, srv1252055430=12, srv1746872516=40, srv386357860=67, srv2121437375=55, srv685497201=81, srv1285869800=14, srv1746492335=39, srv1314683193=16, srv1142196637=7, srv1731050689=38, srv1188658396=10, srv265346526=58, srv11493732=8, srv862013239=89, srv1860304014=44, srv340043714=62, srv1721506478=36, srv748031309=87, srv713817160=83, srv244852309=57, srv36102291=63, srv1827128367=43, srv523253324=71, srv72524773=85, srv477655458=70, srv1528055168=25, srv1114951062=3, srv1462063112=22, srv898435715=92, srv970829622=99, srv911544909=93, srv1798857200=42, srv1940882407=47, srv1941155225=48, srv7162090=84, srv1334877849=19, srv1469441281=23, srv364803077=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-26 23:56:08,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-26 23:56:08,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-26 23:56:08,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-26 23:56:08,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-26 23:56:08,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-26 23:56:08,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-26 23:56:08,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-26 23:56:08,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-26 23:56:08,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-26 23:56:08,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-26 23:56:08,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-26 23:56:08,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-26 23:56:08,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-26 23:56:08,135 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:08,136 WARN [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart) 2023-03-26 23:56:08,136 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.594063860639548E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=5000 2023-03-26 23:56:08,352 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 311 ms to try 5000 different iterations. Found a solution that moves 23 regions; Going from a computed imbalance of 2.594063860639548E-5 to a new imbalance of 7.234288738508196E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0025555555555555557); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:08,352 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1129598405:83 , srv1252055430:83 , srv1469441281:83 , srv1556540352:83 , srv1576553755:83 , srv1641595621:83 , srv1860304014:83 , srv1940882407:83 , srv36102291:83 , srv1029819848:84 , srv103633515:84 , srv1063687939:84 , srv1114951062:84 , srv1129940077:84 , srv1136497730:84 , srv1142196637:84 , srv11493732:84 , srv1188656362:84 , srv1221962339:84 , srv1284957953:84 , srv1285869800:84 , srv1314353267:84 , srv1314683193:84 , srv1322587707:84 , srv1332809676:84 , srv1387818344:84 , srv142900490:84 , srv1462063112:84 , srv1488021782:84 , srv1528055168:84 , srv1565563239:84 , srv1643789669:84 , srv1684704928:84 , srv1698895345:84 , srv1699459974:84 , srv1721506478:84 , srv1727642323:84 , srv1731050689:84 , srv1746492335:84 , srv1746872516:84 , srv1792589038:84 , srv1827128367:84 , srv1865799478:84 , srv1904088499:84 , srv1941155225:84 , srv1948889682:84 , srv1971506394:84 , srv2014991335:84 , srv2084071354:84 , srv2107897377:84 , srv2121437375:84 , srv2144651147:84 , srv244852309:84 , srv265346526:84 , srv27779793:84 , srv2923149:84 , srv322062264:84 , srv340043714:84 , srv362783679:84 , srv364803077:84 , srv378527039:84 , srv422913317:84 , srv428174220:84 , srv54506157:84 , srv553385763:84 , srv564266418:84 , srv591844348:84 , srv592709363:84 , srv613665936:84 , srv63487246:84 , srv641435057:84 , srv684711910:84 , srv685497201:84 , srv713817160:84 , srv7162090:84 , srv72524773:84 , srv748031309:84 , srv813302350:84 , srv862013239:84 , srv865193259:84 , srv891272449:84 , srv898435715:84 , srv911544909:84 , srv911839344:84 , srv915542388:84 , srv924872609:84 , srv94660692:84 , srv948374757:84 , srv970829622:84 , srv1188658396:85 , srv1334877849:85 , srv1554669260:85 , srv1589528626:85 , srv1798857200:85 , srv1988803963:85 , srv386357860:85 , srv477655458:85 , srv523253324:85 , srv728232652:85 , srv700315327:683 } 2023-03-26 23:56:08,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-26 23:56:08,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1314353267=15, srv1641595621=31, srv1322587707=17, srv813302350=88, srv1556540352=27, srv1589528626=30, srv591844348=75, srv1332809676=18, srv1643789669=32, srv1971506394=50, srv362783679=64, srv1129940077=5, srv94660692=97, srv1063687939=2, srv2107897377=54, srv641435057=79, srv1554669260=26, srv1948889682=49, srv428174220=69, srv1684704928=33, srv1904088499=46, srv1565563239=28, srv1488021782=24, srv1792589038=41, srv728232652=86, srv1727642323=37, srv1865799478=45, srv63487246=78, srv1136497730=6, srv564266418=74, srv1029819848=0, srv891272449=91, srv553385763=73, srv1387818344=20, srv322062264=61, srv1284957953=13, srv1988803963=51, srv1698895345=34, srv378527039=66, srv865193259=90, srv1221962339=11, srv1129598405=4, srv592709363=76, srv1699459974=35, srv27779793=59, srv2923149=60, srv54506157=72, srv915542388=95, srv142900490=21, srv911839344=94, srv700315327=82, srv948374757=98, srv613665936=77, srv2014991335=52, srv924872609=96, srv103633515=1, srv1188656362=9, srv422913317=68, srv1576553755=29, srv2084071354=53, srv2144651147=56, srv684711910=80, srv1252055430=12, srv1746872516=40, srv386357860=67, srv2121437375=55, srv685497201=81, srv1285869800=14, srv1746492335=39, srv1314683193=16, srv1142196637=7, srv1731050689=38, srv1188658396=10, srv265346526=58, srv11493732=8, srv862013239=89, srv1860304014=44, srv340043714=62, srv1721506478=36, srv748031309=87, srv713817160=83, srv244852309=57, srv36102291=63, srv1827128367=43, srv523253324=71, srv72524773=85, srv477655458=70, srv1528055168=25, srv1114951062=3, srv1462063112=22, srv898435715=92, srv970829622=99, srv911544909=93, srv1798857200=42, srv1940882407=47, srv1941155225=48, srv7162090=84, srv1334877849=19, srv1469441281=23, srv364803077=65} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-26 23:56:08,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-26 23:56:08,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-26 23:56:08,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-26 23:56:08,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44 2023-03-26 23:56:08,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66 2023-03-26 23:56:08,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87 2023-03-26 23:56:08,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99 2023-03-26 23:56:08,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4 2023-03-26 23:56:08,471 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:08,471 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1129598405:83 , srv1252055430:83 , srv1469441281:83 , srv1556540352:83 , srv1576553755:83 , srv1641595621:83 , srv1860304014:83 , srv1940882407:83 , srv36102291:83 , srv1029819848:84 , srv103633515:84 , srv1063687939:84 , srv1114951062:84 , srv1129940077:84 , srv1136497730:84 , srv1142196637:84 , srv11493732:84 , srv1188656362:84 , srv1221962339:84 , srv1284957953:84 , srv1285869800:84 , srv1314353267:84 , srv1314683193:84 , srv1322587707:84 , srv1332809676:84 , srv1387818344:84 , srv142900490:84 , srv1462063112:84 , srv1488021782:84 , srv1528055168:84 , srv1565563239:84 , srv1643789669:84 , srv1684704928:84 , srv1698895345:84 , srv1699459974:84 , srv1721506478:84 , srv1727642323:84 , srv1731050689:84 , srv1746492335:84 , srv1746872516:84 , srv1792589038:84 , srv1827128367:84 , srv1865799478:84 , srv1904088499:84 , srv1941155225:84 , srv1948889682:84 , srv1971506394:84 , srv2014991335:84 , srv2084071354:84 , srv2107897377:84 , srv2121437375:84 , srv2144651147:84 , srv244852309:84 , srv265346526:84 , srv27779793:84 , srv2923149:84 , srv322062264:84 , srv340043714:84 , srv362783679:84 , srv364803077:84 , srv378527039:84 , srv422913317:84 , srv428174220:84 , srv54506157:84 , srv553385763:84 , srv564266418:84 , srv591844348:84 , srv592709363:84 , srv613665936:84 , srv63487246:84 , srv641435057:84 , srv684711910:84 , srv685497201:84 , srv713817160:84 , srv7162090:84 , srv72524773:84 , srv748031309:84 , srv813302350:84 , srv862013239:84 , srv865193259:84 , srv891272449:84 , srv898435715:84 , srv911544909:84 , srv911839344:84 , srv915542388:84 , srv924872609:84 , srv94660692:84 , srv948374757:84 , srv970829622:84 , srv1188658396:85 , srv1334877849:85 , srv1554669260:85 , srv1589528626:85 , srv1798857200:85 , srv1988803963:85 , srv386357860:85 , srv477655458:85 , srv523253324:85 , srv728232652:85 , srv700315327:683 } 2023-03-26 23:56:08,496 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=441 (was 441), ProcessCount=170 (was 170), AvailableMemoryMB=8967 (was 9450) 2023-03-26 23:56:08,504 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=441, ProcessCount=170, AvailableMemoryMB=8967 2023-03-26 23:56:08,504 INFO [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false 2023-03-26 23:56:08,504 INFO [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc. 2023-03-26 23:56:08,513 INFO [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1070920852:84 , srv1086628225:84 , srv1167354548:84 , srv1326347872:84 , srv142341759:84 , srv1545856199:84 , srv1576123449:84 , srv1761300922:84 , srv1841462719:84 , srv1863259398:84 , srv1894049140:84 , srv1988298679:84 , srv2012344274:84 , srv2049820904:84 , srv2054619024:84 , srv218551798:84 , srv385526308:84 , srv407603186:84 , srv483795872:84 , srv618507687:84 , srv64717537:84 , srv716640443:84 , srv77278717:84 , srv803648442:84 , srv808320045:84 , srv808659685:84 , srv838866156:84 , srv853540228:84 , srv980341931:84 , srv1970268140:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90] 2023-03-26 23:56:08,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-26 23:56:08,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1070920852=0, srv1576123449=6, srv77278717=23, srv1761300922=7, srv1841462719=8, srv1863259398=9, srv142341759=4, srv2054619024=15, srv407603186=18, srv2012344274=13, srv1086628225=1, srv1988298679=12, srv1326347872=3, srv218551798=16, srv1545856199=5, srv980341931=29, srv483795872=19, srv808320045=25, srv618507687=20, srv385526308=17, srv1970268140=11, srv716640443=22, srv838866156=27, srv853540228=28, srv2049820904=14, srv1167354548=2, srv808659685=26, srv803648442=24, srv64717537=21, srv1894049140=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-26 23:56:08,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-26 23:56:08,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-26 23:56:08,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-26 23:56:08,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-26 23:56:08,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-26 23:56:08,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-26 23:56:08,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-26 23:56:08,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-26 23:56:08,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-26 23:56:08,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1520585605037599, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:56:08,538 INFO [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912672174823257, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1520585605037599, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); computedMaxSteps=64800000 2023-03-26 23:58:08,527 INFO [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6051261 different iterations. Found a solution that moves 1845 regions; Going from a computed imbalance of 0.9912672174823257 to a new imbalance of 4.306322040866546E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6833333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:58:08,534 INFO [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1070920852:90 , srv1086628225:90 , srv1167354548:90 , srv1326347872:90 , srv142341759:90 , srv1545856199:90 , srv1576123449:90 , srv1761300922:90 , srv1841462719:90 , srv1863259398:90 , srv1894049140:90 , srv1970268140:90 , srv1988298679:90 , srv2012344274:90 , srv2049820904:90 , srv2054619024:90 , srv218551798:90 , srv385526308:90 , srv407603186:90 , srv483795872:90 , srv618507687:90 , srv64717537:90 , srv716640443:90 , srv77278717:90 , srv803648442:90 , srv808320045:90 , srv808659685:90 , srv838866156:90 , srv853540228:90 , srv980341931:90 } 2023-03-26 23:58:08,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster. 2023-03-26 23:58:08,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1070920852=0, srv1576123449=6, srv77278717=23, srv1761300922=7, srv1841462719=8, srv1863259398=9, srv142341759=4, srv2054619024=15, srv407603186=18, srv2012344274=13, srv1086628225=1, srv1988298679=12, srv1326347872=3, srv218551798=16, srv1545856199=5, srv980341931=29, srv483795872=19, srv808320045=25, srv618507687=20, srv385526308=17, srv1970268140=11, srv716640443=22, srv838866156=27, srv853540228=28, srv2049820904=14, srv1167354548=2, srv808659685=26, srv803648442=24, srv64717537=21, srv1894049140=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2} 2023-03-26 23:58:08,541 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11 2023-03-26 23:58:08,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28 2023-03-26 23:58:08,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29 2023-03-26 23:58:08,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4 2023-03-26 23:58:08,555 INFO [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 2023-03-26 23:58:08,555 INFO [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1070920852:90 , srv1086628225:90 , srv1167354548:90 , srv1326347872:90 , srv142341759:90 , srv1545856199:90 , srv1576123449:90 , srv1761300922:90 , srv1841462719:90 , srv1863259398:90 , srv1894049140:90 , srv1970268140:90 , srv1988298679:90 , srv2012344274:90 , srv2049820904:90 , srv2054619024:90 , srv218551798:90 , srv385526308:90 , srv407603186:90 , srv483795872:90 , srv618507687:90 , srv64717537:90 , srv716640443:90 , srv77278717:90 , srv803648442:90 , srv808320045:90 , srv808659685:90 , srv838866156:90 , srv853540228:90 , srv980341931:90 } 2023-03-26 23:58:08,567 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=295 (was 441), ProcessCount=168 (was 170), AvailableMemoryMB=8907 (was 8967)