2023-10-19 18:56:30,752 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.io.hfile.TestHFile timeout: 13 mins 2023-10-19 18:56:31,079 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3b7dbe10-d52c-7e53-4713-98f3e8aa9805 2023-10-19 18:56:31,247 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-10-19 18:56:31,737 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-10-19 18:56:31,768 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testEmptyHFile Thread=11, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10661 2023-10-19 18:56:31,797 DEBUG [Time-limited test] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-10-19 18:56:31,932 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-10-19 18:56:32,030 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-10-19 18:56:32,079 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testEmptyHFile 2023-10-19 18:56:32,376 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testEmptyHFile Thread=13 (was 11) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=264 (was 263) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10566 (was 10661) 2023-10-19 18:56:32,386 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=13, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10564 2023-10-19 18:56:32,416 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-10-19 18:56:32,445 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on b886434f803f429cb0b647b8cbfddd81 2023-10-19 18:56:32,564 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=13 (was 13), OpenFileDescriptor=265 (was 264) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10549 (was 10564) 2023-10-19 18:56:32,574 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=13, OpenFileDescriptor=265, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10548 2023-10-19 18:56:32,595 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-10-19 18:56:32,617 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 9f36809508924d94aa1ecd66f26701b1 2023-10-19 18:56:32,658 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-10-19 18:56:32,681 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2023-10-19 18:56:32,705 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2023-10-19 18:56:32,717 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2023-10-19 18:56:32,718 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2023-10-19 18:56:32,737 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2023-10-19 18:56:32,737 INFO [Time-limited test] bucket.BucketCache(1318): Shutdown bucket cache: IO persistent=false; path to write=null 2023-10-19 18:56:32,737 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2023-10-19 18:56:32,737 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2023-10-19 18:56:32,750 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=13 (was 13), OpenFileDescriptor=265 (was 265), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10505 (was 10548) 2023-10-19 18:56:32,758 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=13, OpenFileDescriptor=265, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10505 2023-10-19 18:56:33,386 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=13 (was 13), OpenFileDescriptor=265 (was 265), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10485 (was 10505) 2023-10-19 18:56:33,402 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=13, OpenFileDescriptor=265, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10484 2023-10-19 18:56:33,405 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testCorruptTruncatedHFile 2023-10-19 18:56:33,437 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=13 (was 13), OpenFileDescriptor=265 (was 265), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10484 (was 10484) 2023-10-19 18:56:33,446 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=13, OpenFileDescriptor=265, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10483 2023-10-19 18:56:33,460 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-10-19 18:56:33,467 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 900fc7b651d049c0bee4e3e9c7eddb70 2023-10-19 18:56:33,488 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-10-19 18:56:33,493 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2023-10-19 18:56:33,524 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2023-10-19 18:56:33,524 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2023-10-19 18:56:33,529 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2023-10-19 18:56:33,534 INFO [Time-limited test] bucket.BucketCache(1318): Shutdown bucket cache: IO persistent=false; path to write=null 2023-10-19 18:56:33,534 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2023-10-19 18:56:33,534 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2023-10-19 18:56:33,536 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2023-10-19 18:56:33,546 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=14 (was 13) Potentially hanging thread: Time-limited test-BucketCacheWriter-2 java.lang.Throwable.fillInStackTrace(Native Method) java.lang.Throwable.fillInStackTrace(Throwable.java:784) java.lang.Throwable.(Throwable.java:251) java.lang.Exception.(Exception.java:54) java.lang.InterruptedException.(InterruptedException.java:57) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getRAMQueueEntries(BucketCache.java:1154) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:967) - Thread LEAK? -, OpenFileDescriptor=265 (was 265), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10447 (was 10483) 2023-10-19 18:56:33,555 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=13, OpenFileDescriptor=265, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10446 2023-10-19 18:56:33,566 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=13 (was 13), OpenFileDescriptor=266 (was 265) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10446 (was 10446) 2023-10-19 18:56:33,574 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=13, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10445 2023-10-19 18:56:33,587 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-10-19 18:56:33,591 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 1c7766e1ea384c2b97ee2967bd41a640 2023-10-19 18:56:33,608 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=13 (was 13), OpenFileDescriptor=266 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10445 (was 10445) 2023-10-19 18:56:33,618 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testMetaBlocks Thread=13, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10443 2023-10-19 18:56:33,684 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testMetaBlocks Thread=13 (was 13), OpenFileDescriptor=266 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10442 (was 10443) 2023-10-19 18:56:33,694 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testGetShortMidpoint Thread=13, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10442 2023-10-19 18:56:33,709 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testGetShortMidpoint Thread=13 (was 13), OpenFileDescriptor=266 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10441 (was 10442) 2023-10-19 18:56:33,719 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCompressionOrdinance Thread=13, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10441 2023-10-19 18:56:33,727 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCompressionOrdinance Thread=13 (was 13), OpenFileDescriptor=266 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10441 (was 10441) 2023-10-19 18:56:33,738 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=13, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10441 2023-10-19 18:56:33,752 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-10-19 18:56:33,757 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on a0e0709f176b4d5985d243bf1f119667 2023-10-19 18:56:33,777 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-10-19 18:56:33,820 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2023-10-19 18:56:33,828 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2023-10-19 18:56:33,829 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2023-10-19 18:56:33,830 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2023-10-19 18:56:33,847 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2023-10-19 18:56:33,847 INFO [Time-limited test] bucket.BucketCache(1318): Shutdown bucket cache: IO persistent=false; path to write=null 2023-10-19 18:56:33,847 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2023-10-19 18:56:33,848 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2023-10-19 18:56:33,860 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=14 (was 13) Potentially hanging thread: ForkJoinPool.commonPool-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) - Thread LEAK? -, OpenFileDescriptor=266 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10403 (was 10441) 2023-10-19 18:56:33,868 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testTFileFeatures Thread=14, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10401 2023-10-19 18:56:33,875 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@7a16e15d, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2023-10-19 18:56:33,894 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@5d88089a, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2023-10-19 18:56:33,913 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@5677fba2, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2023-10-19 18:56:33,926 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@1ee0519a, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2023-10-19 18:56:33,946 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testTFileFeatures Thread=14 (was 14), OpenFileDescriptor=266 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10399 (was 10401) 2023-10-19 18:56:33,955 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testDBEShipped Thread=14, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10399 2023-10-19 18:56:33,966 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_PREFIX 2023-10-19 18:56:33,981 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_DIFF 2023-10-19 18:56:33,994 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_FAST_DIFF 2023-10-19 18:56:33,999 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_ROW_INDEX_V1 2023-10-19 18:56:34,009 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testDBEShipped Thread=14 (was 14), OpenFileDescriptor=266 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10397 (was 10399) 2023-10-19 18:56:34,018 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testShortMidpointSameQual Thread=14, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10397 2023-10-19 18:56:34,026 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testShortMidpointSameQual Thread=14 (was 14), OpenFileDescriptor=266 (was 266), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10397 (was 10397) 2023-10-19 18:56:34,039 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testNullMetaBlocks Thread=14, OpenFileDescriptor=266, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10396 2023-10-19 18:56:34,061 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testNullMetaBlocks Thread=14 (was 14), OpenFileDescriptor=268 (was 266) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10393 (was 10396) 2023-10-19 18:56:34,068 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=14, OpenFileDescriptor=268, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=168, AvailableMemoryMB=10393 2023-10-19 18:56:34,083 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-10-19 18:56:34,102 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on f220bada466749e0a56c87c94c6ff654 2023-10-19 18:56:34,122 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-10-19 18:56:34,128 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2023-10-19 18:56:34,163 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2023-10-19 18:56:34,166 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2023-10-19 18:56:34,169 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2023-10-19 18:56:34,173 INFO [Time-limited test] bucket.BucketCache(1318): Shutdown bucket cache: IO persistent=false; path to write=null 2023-10-19 18:56:34,174 ERROR [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(975): WriterThread encountered error java.lang.IllegalArgumentException: ReferenceCount : 0 (expected: > 0) at org.apache.hbase.thirdparty.io.netty.util.internal.ObjectUtil.checkPositive(ObjectUtil.java:100) at org.apache.hadoop.hbase.nio.ByteBuff.checkRefCount(ByteBuff.java:77) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:683) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:43) at org.apache.hadoop.hbase.io.hfile.HFileBlock.getBufferReadOnly(HFileBlock.java:506) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1503) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1059) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:973) 2023-10-19 18:56:34,175 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2023-10-19 18:56:34,175 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2023-10-19 18:56:34,178 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2023-10-19 18:56:34,184 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=18 (was 14) Potentially hanging thread: Time-limited test-BucketCacheWriter-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getRAMQueueEntries(BucketCache.java:1154) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:967) Potentially hanging thread: LruBlockCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Time-limited test-BucketCacheWriter-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getRAMQueueEntries(BucketCache.java:1154) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:967) Potentially hanging thread: Time-limited test-BucketCacheWriter-2 java.lang.Throwable.printStackTrace(Throwable.java:722) org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:59) org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:84) org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:417) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:306) org.apache.log4j.WriterAppender.append(WriterAppender.java:156) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:232) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:57) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.error(Reload4jLoggerAdapter.java:555) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:975) - Thread LEAK? -, OpenFileDescriptor=265 (was 268), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 373), ProcessCount=168 (was 168), AvailableMemoryMB=10364 (was 10393)