2024-03-02 18:53:18,379 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.region.TestMasterRegionWALCleaner timeout: 13 mins 2024-03-02 18:53:18,417 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.region.TestMasterRegionWALCleaner#test Thread=9, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=323, ProcessCount=172, AvailableMemoryMB=10210 2024-03-02 18:53:18,449 INFO [Time-limited test] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2024-03-02 18:53:18,450 INFO [Time-limited test] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2024-03-02 18:53:19,134 INFO [Time-limited test] region.MasterRegion(309): Create or load local region for table 'test:local', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2024-03-02 18:53:19,257 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-03-02 18:53:19,711 INFO [Time-limited test] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2024-03-02 18:53:19,932 DEBUG [Time-limited test] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2024-03-02 18:53:19,932 DEBUG [Time-limited test] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2024-03-02 18:53:19,935 DEBUG [Time-limited test] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionTestBase.createMasterRegion(MasterRegionTestBase.java:109) at org.apache.hadoop.hbase.master.region.MasterRegionTestBase.setUp(MasterRegionTestBase.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2024-03-02 18:53:19,943 DEBUG [Time-limited test] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionTestBase.createMasterRegion(MasterRegionTestBase.java:109) at org.apache.hadoop.hbase.master.region.MasterRegionTestBase.setUp(MasterRegionTestBase.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2024-03-02 18:53:19,949 INFO [Time-limited test] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2024-03-02 18:53:20,120 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-03-02 18:53:20,237 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-03-02 18:53:20,313 DEBUG [Time-limited test] util.ClassSize(228): Using Unsafe to estimate memory layout 2024-03-02 18:53:20,327 INFO [Time-limited test] regionserver.HRegion(7693): Creating {ENCODED => fb3251d5e68787855bc0679a918ea02e, NAME => 'test:local,,1.fb3251d5e68787855bc0679a918ea02e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='test:local', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local-tmp 2024-03-02 18:53:20,456 DEBUG [Time-limited test] regionserver.HRegion(866): Instantiated test:local,,1.fb3251d5e68787855bc0679a918ea02e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2024-03-02 18:53:20,586 DEBUG [Time-limited test] regionserver.HRegion(1604): Closing fb3251d5e68787855bc0679a918ea02e, disabling compactions & flushes 2024-03-02 18:53:20,587 INFO [Time-limited test] regionserver.HRegion(1626): Closing region test:local,,1.fb3251d5e68787855bc0679a918ea02e. 2024-03-02 18:53:20,587 DEBUG [Time-limited test] regionserver.HRegion(1647): Waiting without time limit for close lock on test:local,,1.fb3251d5e68787855bc0679a918ea02e. 2024-03-02 18:53:20,588 DEBUG [Time-limited test] regionserver.HRegion(1714): Acquired close lock on test:local,,1.fb3251d5e68787855bc0679a918ea02e. after waiting 0 ms 2024-03-02 18:53:20,588 DEBUG [Time-limited test] regionserver.HRegion(1724): Updates disabled for region test:local,,1.fb3251d5e68787855bc0679a918ea02e. 2024-03-02 18:53:20,588 INFO [Time-limited test] regionserver.HRegion(1838): Closed test:local,,1.fb3251d5e68787855bc0679a918ea02e. 2024-03-02 18:53:20,589 DEBUG [Time-limited test] regionserver.HRegion(1558): Region close journal for fb3251d5e68787855bc0679a918ea02e: 2024-03-02 18:53:20,591 DEBUG [Time-limited test] region.MasterRegion(191): WALDir=file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/WALs/localhost,12345,1709405599102 2024-03-02 18:53:20,626 INFO [Time-limited test] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=localhost%2C12345%2C1709405599102, suffix=, logDir=file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/WALs/localhost,12345,1709405599102, archiveDir=file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/oldWALs, maxLogs=32 2024-03-02 18:53:20,690 INFO [Time-limited test] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/WALs/localhost,12345,1709405599102/localhost%2C12345%2C1709405599102.1709405600640 2024-03-02 18:53:20,690 DEBUG [Time-limited test] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [] 2024-03-02 18:53:20,691 DEBUG [Time-limited test] regionserver.HRegion(7854): Opening region: {ENCODED => fb3251d5e68787855bc0679a918ea02e, NAME => 'test:local,,1.fb3251d5e68787855bc0679a918ea02e.', STARTKEY => '', ENDKEY => ''} 2024-03-02 18:53:20,692 DEBUG [Time-limited test] regionserver.HRegion(866): Instantiated test:local,,1.fb3251d5e68787855bc0679a918ea02e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2024-03-02 18:53:20,695 DEBUG [Time-limited test] regionserver.HRegion(7894): checking encryption for fb3251d5e68787855bc0679a918ea02e 2024-03-02 18:53:20,696 DEBUG [Time-limited test] regionserver.HRegion(7897): checking classloading for fb3251d5e68787855bc0679a918ea02e 2024-03-02 18:53:20,754 INFO [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region fb3251d5e68787855bc0679a918ea02e 2024-03-02 18:53:20,756 WARN [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] util.CommonFSUtils(529): Unable to set storagePolicy=HOT for path=file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e/f1. DEBUG log level might have more details. java.lang.UnsupportedOperationException: RawLocalFileSystem doesn't support setStoragePolicy at org.apache.hadoop.fs.FileSystem.setStoragePolicy(FileSystem.java:3041) at org.apache.hadoop.fs.FilterFileSystem.setStoragePolicy(FilterFileSystem.java:643) at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:521) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:504) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:477) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:220) at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:252) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6371) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1117) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-03-02 18:53:20,756 DEBUG [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] util.CommonFSUtils(539): The underlying FileSystem implementation doesn't support setStoragePolicy. This is probably intentional on their part, since HDFS-9345 appears to be present in your version of Hadoop. For more information check the Hadoop documentation on 'ArchivalStorage', the Hadoop FileSystem specification docs from HADOOP-11981, and/or related documentation from the provider of the underlying FileSystem (its name should appear in the stacktrace that accompanies this message). Note in particular that Hadoop's local filesystem implementation doesn't support storage policies. java.lang.UnsupportedOperationException: RawLocalFileSystem doesn't support setStoragePolicy at org.apache.hadoop.fs.FileSystem.setStoragePolicy(FileSystem.java:3041) at org.apache.hadoop.fs.FilterFileSystem.setStoragePolicy(FilterFileSystem.java:643) at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:521) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:504) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:477) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:220) at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:252) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6371) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1117) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-03-02 18:53:20,780 INFO [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb3251d5e68787855bc0679a918ea02e columnFamilyName f1 2024-03-02 18:53:20,797 INFO [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] regionserver.HStore(310): Store=fb3251d5e68787855bc0679a918ea02e/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2024-03-02 18:53:20,797 INFO [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region fb3251d5e68787855bc0679a918ea02e 2024-03-02 18:53:20,798 DEBUG [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] util.CommonFSUtils(532): Unable to set storagePolicy=HOT for path=file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e/f2 java.lang.UnsupportedOperationException: RawLocalFileSystem doesn't support setStoragePolicy at org.apache.hadoop.fs.FileSystem.setStoragePolicy(FileSystem.java:3041) at org.apache.hadoop.fs.FilterFileSystem.setStoragePolicy(FilterFileSystem.java:643) at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:521) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:504) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:477) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:220) at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:252) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6371) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1117) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-03-02 18:53:20,798 DEBUG [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] util.CommonFSUtils(539): The underlying FileSystem implementation doesn't support setStoragePolicy. This is probably intentional on their part, since HDFS-9345 appears to be present in your version of Hadoop. For more information check the Hadoop documentation on 'ArchivalStorage', the Hadoop FileSystem specification docs from HADOOP-11981, and/or related documentation from the provider of the underlying FileSystem (its name should appear in the stacktrace that accompanies this message). Note in particular that Hadoop's local filesystem implementation doesn't support storage policies. java.lang.UnsupportedOperationException: RawLocalFileSystem doesn't support setStoragePolicy at org.apache.hadoop.fs.FileSystem.setStoragePolicy(FileSystem.java:3041) at org.apache.hadoop.fs.FilterFileSystem.setStoragePolicy(FilterFileSystem.java:643) at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:521) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:504) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:477) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:220) at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:252) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6371) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1117) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2024-03-02 18:53:20,799 INFO [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fb3251d5e68787855bc0679a918ea02e columnFamilyName f2 2024-03-02 18:53:20,800 INFO [StoreOpener-fb3251d5e68787855bc0679a918ea02e-1] regionserver.HStore(310): Store=fb3251d5e68787855bc0679a918ea02e/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2024-03-02 18:53:20,806 DEBUG [Time-limited test] regionserver.HRegion(5209): Found 0 recovered edits file(s) under file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e 2024-03-02 18:53:20,807 DEBUG [Time-limited test] regionserver.HRegion(5209): Found 0 recovered edits file(s) under file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e 2024-03-02 18:53:20,827 DEBUG [Time-limited test] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table test:local descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2024-03-02 18:53:20,831 DEBUG [Time-limited test] regionserver.HRegion(1055): writing seq id for fb3251d5e68787855bc0679a918ea02e 2024-03-02 18:53:20,848 DEBUG [Time-limited test] wal.WALSplitUtil(408): Wrote file=file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2024-03-02 18:53:20,849 INFO [Time-limited test] regionserver.HRegion(1072): Opened fb3251d5e68787855bc0679a918ea02e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63641893, jitterRate=-0.0516618937253952}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2024-03-02 18:53:20,849 DEBUG [Time-limited test] regionserver.HRegion(965): Region open journal for fb3251d5e68787855bc0679a918ea02e: 2024-03-02 18:53:20,851 INFO [Time-limited test] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2024-03-02 18:53:20,868 INFO [Time-limited test] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2024-03-02 18:53:20,869 INFO [Time-limited test] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2024-03-02 18:53:20,869 INFO [Time-limited test] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2024-03-02 18:53:20,870 INFO [Time-limited test] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2024-03-02 18:53:20,870 INFO [Time-limited test] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=1000, unit=MILLISECONDS is enabled. 2024-03-02 18:53:20,871 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 0 ms. 2024-03-02 18:53:20,925 INFO [Time-limited test] regionserver.HRegion(2745): Flushing fb3251d5e68787855bc0679a918ea02e 2/2 column families, dataSize=31 B heapSize=632 B 2024-03-02 18:53:20,992 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 8a56281d89304eadb6dcdef8190ef173 2024-03-02 18:53:21,050 INFO [Time-limited test] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31 B at sequenceid=4 (bloomFilter=true), to=file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e/.tmp/f1/8a56281d89304eadb6dcdef8190ef173 2024-03-02 18:53:21,100 DEBUG [Time-limited test] regionserver.HRegionFileSystem(485): Committing file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e/.tmp/f1/8a56281d89304eadb6dcdef8190ef173 as file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e/f1/8a56281d89304eadb6dcdef8190ef173 2024-03-02 18:53:21,103 INFO [Time-limited test] regionserver.HStore(1080): Added file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/data/test/local/fb3251d5e68787855bc0679a918ea02e/f1/8a56281d89304eadb6dcdef8190ef173, entries=1, sequenceid=4, filesize=4.8 K 2024-03-02 18:53:21,105 INFO [Time-limited test] regionserver.HRegion(2948): Finished flush of dataSize ~31 B/31, heapSize ~360 B/360, currentSize=0 B/0 for fb3251d5e68787855bc0679a918ea02e in 180ms, sequenceid=4, compaction requested=false 2024-03-02 18:53:21,106 DEBUG [Time-limited test] regionserver.HRegion(2446): Flush status journal for fb3251d5e68787855bc0679a918ea02e: 2024-03-02 18:53:21,107 DEBUG [test:local-WAL-Roller] wal.AbstractWALRoller(197): WAL AsyncFSWAL localhost%2C12345%2C1709405599102:(num 1709405600640) roll requested 2024-03-02 18:53:21,130 INFO [test:local-WAL-Roller] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/WALs/localhost,12345,1709405599102/localhost%2C12345%2C1709405599102.1709405600640 with entries=3, filesize=658 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/WALs/localhost,12345,1709405599102/localhost%2C12345%2C1709405599102.1709405601107 2024-03-02 18:53:21,131 DEBUG [test:local-WAL-Roller] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [] 2024-03-02 18:53:21,132 DEBUG [test:local-WAL-Roller] wal.AbstractFSWAL(716): file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/WALs/localhost,12345,1709405599102/localhost%2C12345%2C1709405599102.1709405600640 is not closed yet, will try archiving it next time 2024-03-02 18:53:21,133 INFO [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/WALs/localhost,12345,1709405599102/localhost%2C12345%2C1709405599102.1709405600640 to file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/oldWALs/localhost%2C12345%2C1709405599102.1709405600640 2024-03-02 18:53:21,137 INFO [WAL-Archive-0] region.MasterRegionUtils(50): Moved file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/local/oldWALs/localhost%2C12345%2C1709405599102.1709405600640 to file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/oldWALs/localhost%2C12345%2C1709405599102.1709405600640$masterlocalwal$ 2024-03-02 18:53:21,880 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 9 ms. 2024-03-02 18:53:22,872 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 1 ms. 2024-03-02 18:53:23,872 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 0 ms. 2024-03-02 18:53:24,872 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 0 ms. 2024-03-02 18:53:25,872 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 0 ms. 2024-03-02 18:53:26,874 DEBUG [log_cleaner-dir-scan-pool-0] cleaner.LogCleaner(121): Old WALs for delete: localhost%2C12345%2C1709405599102.1709405600640$masterlocalwal$ 2024-03-02 18:53:26,874 DEBUG [OldWALsCleaner-0] cleaner.LogCleaner(176): Deleting DeprecatedRawLocalFileStatus{path=file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/894d70dc-270d-477d-5747-2301a5200683/oldWALs/localhost%2C12345%2C1709405599102.1709405600640$masterlocalwal$; isDirectory=false; length=666; replication=1; blocksize=33554432; modification_time=1709405601134; access_time=1709405600653; owner=; group=; permission=rw-rw-rw-; isSymlink=false} 2024-03-02 18:53:26,875 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 4 ms. 2024-03-02 18:53:27,871 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 0 ms. 2024-03-02 18:53:28,871 DEBUG [TestMasterRegionWALCleaner.Chore.1] hbase.ScheduledChore(162): LogsCleaner execution time: 0 ms. 2024-03-02 18:53:29,208 INFO [Time-limited test] region.MasterRegion(167): Closing local region {ENCODED => fb3251d5e68787855bc0679a918ea02e, NAME => 'test:local,,1.fb3251d5e68787855bc0679a918ea02e.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2024-03-02 18:53:29,211 DEBUG [Time-limited test] regionserver.HRegion(1604): Closing fb3251d5e68787855bc0679a918ea02e, disabling compactions & flushes 2024-03-02 18:53:29,211 INFO [Time-limited test] regionserver.HRegion(1626): Closing region test:local,,1.fb3251d5e68787855bc0679a918ea02e. 2024-03-02 18:53:29,211 DEBUG [Time-limited test] regionserver.HRegion(1647): Waiting without time limit for close lock on test:local,,1.fb3251d5e68787855bc0679a918ea02e. 2024-03-02 18:53:29,211 DEBUG [Time-limited test] regionserver.HRegion(1714): Acquired close lock on test:local,,1.fb3251d5e68787855bc0679a918ea02e. after waiting 0 ms 2024-03-02 18:53:29,212 DEBUG [Time-limited test] regionserver.HRegion(1724): Updates disabled for region test:local,,1.fb3251d5e68787855bc0679a918ea02e. 2024-03-02 18:53:29,214 INFO [Time-limited test] regionserver.HRegion(1838): Closed test:local,,1.fb3251d5e68787855bc0679a918ea02e. 2024-03-02 18:53:29,214 DEBUG [Time-limited test] regionserver.HRegion(1558): Region close journal for fb3251d5e68787855bc0679a918ea02e: 2024-03-02 18:53:29,214 INFO [test:local-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2024-03-02 18:53:29,215 INFO [Time-limited test] hbase.ChoreService(369): Chore service for: TestMasterRegionWALCleaner had [ScheduledChore name=LogsCleaner, period=1000, unit=MILLISECONDS] on shutdown 2024-03-02 18:53:29,248 INFO [Time-limited test] hbase.ResourceChecker(175): after: master.region.TestMasterRegionWALCleaner#test Thread=15 (was 9) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:327) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: OldWALsCleaner-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:172) org.apache.hadoop.hbase.master.cleaner.LogCleaner.lambda$createOldWalsCleaner$1(LogCleaner.java:152) org.apache.hadoop.hbase.master.cleaner.LogCleaner$$Lambda$117/1966950998.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=294 (was 290) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=289 (was 323), ProcessCount=172 (was 172), AvailableMemoryMB=9285 (was 10210)