2024-01-30 04:55:29,418 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.io.hfile.TestHFile timeout: 13 mins 2024-01-30 04:55:29,612 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9fddd05d-61fd-a158-cd17-6fe10e7af1a4 2024-01-30 04:55:29,743 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-01-30 04:55:30,116 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2024-01-30 04:55:30,137 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testEmptyHFile Thread=11, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=3357 2024-01-30 04:55:30,164 DEBUG [Time-limited test] util.ClassSize(228): Using Unsafe to estimate memory layout 2024-01-30 04:55:30,288 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-30 04:55:30,384 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-30 04:55:30,417 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testEmptyHFile 2024-01-30 04:55:30,651 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testEmptyHFile Thread=13 (was 11) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=291 (was 290) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=3199 (was 3357) 2024-01-30 04:55:30,658 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=13, OpenFileDescriptor=291, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=3197 2024-01-30 04:55:30,679 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-30 04:55:30,707 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on e18d258651ee4921af181890796dbf47 2024-01-30 04:55:30,795 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=13 (was 13), OpenFileDescriptor=292 (was 291) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=3165 (was 3197) 2024-01-30 04:55:30,805 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=3163 2024-01-30 04:55:30,819 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-30 04:55:30,832 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 45cbae074ec1442b846124c2948d0911 2024-01-30 04:55:30,856 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-30 04:55:30,869 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-30 04:55:30,878 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-30 04:55:30,891 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-30 04:55:30,892 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-30 04:55:30,900 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 45cbae074ec1442b846124c2948d0911_116156 had block type DATA, but was found in L1 cache. 2024-01-30 04:55:30,907 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-30 04:55:30,907 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-30 04:55:30,907 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-30 04:55:30,907 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-30 04:55:30,918 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=3098 (was 3163) 2024-01-30 04:55:30,925 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=3096 2024-01-30 04:55:31,514 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2972 (was 3096) 2024-01-30 04:55:31,533 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2959 2024-01-30 04:55:31,538 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testCorruptTruncatedHFile 2024-01-30 04:55:31,565 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2956 (was 2959) 2024-01-30 04:55:31,572 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2955 2024-01-30 04:55:31,590 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-30 04:55:31,596 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on da45e6dbf987451288a98a6b9ec0c464 2024-01-30 04:55:31,620 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-30 04:55:31,624 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-30 04:55:31,639 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-30 04:55:31,639 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-30 04:55:31,640 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-30 04:55:31,643 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key da45e6dbf987451288a98a6b9ec0c464_115789 had block type DATA, but was found in L1 cache. 2024-01-30 04:55:31,643 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-30 04:55:31,643 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-30 04:55:31,643 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-30 04:55:31,643 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-30 04:55:31,652 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2899 (was 2955) 2024-01-30 04:55:31,658 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2897 2024-01-30 04:55:31,668 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=13 (was 13), OpenFileDescriptor=293 (was 292) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2895 (was 2897) 2024-01-30 04:55:31,674 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2893 2024-01-30 04:55:31,687 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-30 04:55:31,692 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on daf396e80d2847f68e22999702715280 2024-01-30 04:55:31,713 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=13 (was 13), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2891 (was 2893) 2024-01-30 04:55:31,720 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testMetaBlocks Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2890 2024-01-30 04:55:31,779 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testMetaBlocks Thread=13 (was 13), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2889 (was 2890) 2024-01-30 04:55:31,789 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testGetShortMidpoint Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2889 2024-01-30 04:55:31,804 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testGetShortMidpoint Thread=13 (was 13), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2888 (was 2889) 2024-01-30 04:55:31,813 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCompressionOrdinance Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2888 2024-01-30 04:55:31,820 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCompressionOrdinance Thread=13 (was 13), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2887 (was 2888) 2024-01-30 04:55:31,829 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2887 2024-01-30 04:55:31,841 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-30 04:55:31,845 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 84a45fbfd5434b9299261bc96da1b8aa 2024-01-30 04:55:31,869 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-30 04:55:31,906 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-30 04:55:31,916 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-30 04:55:31,916 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-30 04:55:31,920 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-30 04:55:31,929 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 84a45fbfd5434b9299261bc96da1b8aa_116137 had block type DATA, but was found in L1 cache. 2024-01-30 04:55:31,934 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-30 04:55:31,934 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-30 04:55:31,934 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-30 04:55:31,935 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-30 04:55:31,943 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=14 (was 13) Potentially hanging thread: ForkJoinPool.commonPool-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) - Thread LEAK? -, OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2852 (was 2887) 2024-01-30 04:55:31,952 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testTFileFeatures Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2852 2024-01-30 04:55:31,955 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@16a9875a, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-30 04:55:31,973 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@63b28ae, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-30 04:55:31,998 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@20cff150, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-30 04:55:32,017 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@63cc7af3, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-30 04:55:32,037 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testTFileFeatures Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2846 (was 2852) 2024-01-30 04:55:32,044 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testDBEShipped Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2845 2024-01-30 04:55:32,048 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_PREFIX 2024-01-30 04:55:32,053 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_DIFF 2024-01-30 04:55:32,057 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_FAST_DIFF 2024-01-30 04:55:32,060 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_ROW_INDEX_V1 2024-01-30 04:55:32,068 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testDBEShipped Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2842 (was 2845) 2024-01-30 04:55:32,074 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testShortMidpointSameQual Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2841 2024-01-30 04:55:32,081 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testShortMidpointSameQual Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2841 (was 2841) 2024-01-30 04:55:32,087 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testNullMetaBlocks Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2841 2024-01-30 04:55:32,104 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testNullMetaBlocks Thread=14 (was 14), OpenFileDescriptor=295 (was 293) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2840 (was 2841) 2024-01-30 04:55:32,113 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=14, OpenFileDescriptor=295, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=167, AvailableMemoryMB=2839 2024-01-30 04:55:32,126 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-30 04:55:32,131 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 2dd96a0811c840ada1954c7ca52455da 2024-01-30 04:55:32,149 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-30 04:55:32,150 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-30 04:55:32,158 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-30 04:55:32,158 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-30 04:55:32,170 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-30 04:55:32,173 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 2dd96a0811c840ada1954c7ca52455da_116549 had block type DATA, but was found in L1 cache. 2024-01-30 04:55:32,174 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-30 04:55:32,174 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-30 04:55:32,174 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-30 04:55:32,176 ERROR [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(974): WriterThread encountered error java.lang.IllegalArgumentException: ReferenceCount : 0 (expected: > 0) at org.apache.hbase.thirdparty.io.netty.util.internal.ObjectUtil.checkPositive(ObjectUtil.java:100) at org.apache.hadoop.hbase.nio.ByteBuff.checkRefCount(ByteBuff.java:77) at org.apache.hadoop.hbase.nio.MultiByteBuff.position(MultiByteBuff.java:396) at org.apache.hadoop.hbase.nio.MultiByteBuff.position(MultiByteBuff.java:43) at org.apache.hadoop.hbase.util.ByteBufferArray.lambda$static$1(ByteBufferArray.java:149) at org.apache.hadoop.hbase.util.ByteBufferArray.internalTransfer(ByteBufferArray.java:180) at org.apache.hadoop.hbase.util.ByteBufferArray.write(ByteBufferArray.java:139) at org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.write(ByteBufferIOEngine.java:127) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1506) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1059) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:972) 2024-01-30 04:55:32,179 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-30 04:55:32,184 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=17 (was 14) Potentially hanging thread: BucketCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LruBlockCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Time-limited test-BucketCacheWriter-1 org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:177) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.error(Reload4jLoggerAdapter.java:555) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:974) - Thread LEAK? -, OpenFileDescriptor=292 (was 295), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=277 (was 277), ProcessCount=167 (was 167), AvailableMemoryMB=2805 (was 2839)