2023-03-27 09:00:42,030 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.io.hfile.TestHFile timeout: 13 mins 2023-03-27 09:00:42,386 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/128cf0af-fa4d-5a32-1c56-04f50a279c3b 2023-03-27 09:00:42,510 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-27 09:00:42,923 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-27 09:00:42,948 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testEmptyHFile Thread=9, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=926, ProcessCount=178, AvailableMemoryMB=3229 2023-03-27 09:00:42,967 DEBUG [Time-limited test] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-27 09:00:43,070 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-27 09:00:43,145 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-27 09:00:43,174 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testEmptyHFile 2023-03-27 09:00:43,461 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testEmptyHFile Thread=11 (was 9) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=261 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=926 (was 926), ProcessCount=178 (was 178), AvailableMemoryMB=3183 (was 3229) 2023-03-27 09:00:43,484 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=11, OpenFileDescriptor=261, MaxFileDescriptor=60000, SystemLoadAverage=926, ProcessCount=178, AvailableMemoryMB=3183 2023-03-27 09:00:43,514 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-03-27 09:00:43,562 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 4c3676f6184045b9819c9371e311ac57 2023-03-27 09:00:43,684 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=11 (was 11), OpenFileDescriptor=262 (was 261) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=926 (was 926), ProcessCount=178 (was 178), AvailableMemoryMB=3177 (was 3183) 2023-03-27 09:00:43,694 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=11, OpenFileDescriptor=262, MaxFileDescriptor=60000, SystemLoadAverage=926, ProcessCount=178, AvailableMemoryMB=3175 2023-03-27 09:00:43,715 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-03-27 09:00:43,734 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 28f65b71af274e4b93751fdecccc1c43 2023-03-27 09:00:43,767 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-27 09:00:43,786 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2023-03-27 09:00:43,801 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2023-03-27 09:00:43,815 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2023-03-27 09:00:43,822 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2023-03-27 09:00:43,842 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2023-03-27 09:00:43,842 INFO [Time-limited test] bucket.BucketCache(1318): Shutdown bucket cache: IO persistent=false; path to write=null 2023-03-27 09:00:43,844 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2023-03-27 09:00:43,846 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2023-03-27 09:00:43,862 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=13 (was 11) Potentially hanging thread: BucketCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LruAdaptiveBlockCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=262 (was 262), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=926 (was 926), ProcessCount=178 (was 178), AvailableMemoryMB=3141 (was 3175) 2023-03-27 09:00:43,879 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=11, OpenFileDescriptor=262, MaxFileDescriptor=60000, SystemLoadAverage=926, ProcessCount=178, AvailableMemoryMB=3141 2023-03-27 09:00:44,706 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=11 (was 11), OpenFileDescriptor=262 (was 262), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=926 (was 926), ProcessCount=178 (was 178), AvailableMemoryMB=3131 (was 3141) 2023-03-27 09:00:44,723 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=11, OpenFileDescriptor=262, MaxFileDescriptor=60000, SystemLoadAverage=926, ProcessCount=178, AvailableMemoryMB=3130 2023-03-27 09:00:44,727 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testCorruptTruncatedHFile 2023-03-27 09:00:44,775 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=11 (was 11), OpenFileDescriptor=262 (was 262), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=926 (was 926), ProcessCount=178 (was 178), AvailableMemoryMB=3128 (was 3130) 2023-03-27 09:00:44,787 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=11, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=926, ProcessCount=178, AvailableMemoryMB=3128 2023-03-27 09:00:44,826 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-03-27 09:00:44,835 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on e8d72589ef9b40f4a031b964e8c3074b 2023-03-27 09:00:44,877 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-27 09:00:44,886 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2023-03-27 09:00:44,912 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2023-03-27 09:00:44,912 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2023-03-27 09:00:44,918 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2023-03-27 09:00:44,923 INFO [Time-limited test] bucket.BucketCache(1318): Shutdown bucket cache: IO persistent=false; path to write=null 2023-03-27 09:00:44,923 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2023-03-27 09:00:44,923 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2023-03-27 09:00:44,926 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2023-03-27 09:00:44,934 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=13 (was 11) Potentially hanging thread: Time-limited test-BucketCacheWriter-2 java.lang.Double.doubleToRawLongBits(Native Method) java.lang.Math.getExponent(Math.java:1840) java.lang.StrictMath.floorOrCeil(StrictMath.java:374) java.lang.StrictMath.floor(StrictMath.java:359) java.lang.Math.floor(Math.java:442) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.acceptableSize(BucketCache.java:752) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1135) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:973) - Thread LEAK? -, OpenFileDescriptor=262 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 926) - SystemLoadAverage LEAK? -, ProcessCount=178 (was 178), AvailableMemoryMB=3115 (was 3128) 2023-03-27 09:00:44,943 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=12, OpenFileDescriptor=262, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3115 2023-03-27 09:00:44,955 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=12 (was 12), OpenFileDescriptor=263 (was 262) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3115 (was 3115) 2023-03-27 09:00:44,964 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3115 2023-03-27 09:00:44,981 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-03-27 09:00:44,985 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on ce1dc7427e6a4cf9a8cf4d4adb469a55 2023-03-27 09:00:45,012 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3114 (was 3115) 2023-03-27 09:00:45,022 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testMetaBlocks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3113 2023-03-27 09:00:45,083 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testMetaBlocks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3113 (was 3113) 2023-03-27 09:00:45,092 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testGetShortMidpoint Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3113 2023-03-27 09:00:45,106 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testGetShortMidpoint Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3113 (was 3113) 2023-03-27 09:00:45,116 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCompressionOrdinance Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3113 2023-03-27 09:00:45,125 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCompressionOrdinance Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3113 (was 3113) 2023-03-27 09:00:45,134 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3113 2023-03-27 09:00:45,149 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-03-27 09:00:45,160 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 31551324c5ce4c5fb8167e4940b5e21c 2023-03-27 09:00:45,193 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-27 09:00:45,228 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2023-03-27 09:00:45,236 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2023-03-27 09:00:45,236 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2023-03-27 09:00:45,238 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2023-03-27 09:00:45,255 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2023-03-27 09:00:45,255 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2023-03-27 09:00:45,255 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2023-03-27 09:00:45,255 INFO [Time-limited test] bucket.BucketCache(1318): Shutdown bucket cache: IO persistent=false; path to write=null 2023-03-27 09:00:45,265 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=14 (was 12) Potentially hanging thread: BucketCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool.commonPool-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) - Thread LEAK? -, OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3089 (was 3113) 2023-03-27 09:00:45,273 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testTFileFeatures Thread=13, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3089 2023-03-27 09:00:45,278 INFO [Time-limited test] hfile.TestHFile(503): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@6465da6, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2023-03-27 09:00:45,298 INFO [Time-limited test] hfile.TestHFile(503): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@5d332f0d, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2023-03-27 09:00:45,338 INFO [Time-limited test] hfile.TestHFile(503): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@1c54eec0, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2023-03-27 09:00:45,362 INFO [Time-limited test] hfile.TestHFile(503): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@23d1f984, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2023-03-27 09:00:45,389 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testTFileFeatures Thread=13 (was 13), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3088 (was 3089) 2023-03-27 09:00:45,402 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testDBEShipped Thread=13, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3087 2023-03-27 09:00:45,407 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_PREFIX 2023-03-27 09:00:45,413 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_DIFF 2023-03-27 09:00:45,416 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_FAST_DIFF 2023-03-27 09:00:45,420 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_ROW_INDEX_V1 2023-03-27 09:00:45,430 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testDBEShipped Thread=13 (was 13), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3087 (was 3087) 2023-03-27 09:00:45,441 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testShortMidpointSameQual Thread=13, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3087 2023-03-27 09:00:45,450 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testShortMidpointSameQual Thread=13 (was 13), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3086 (was 3087) 2023-03-27 09:00:45,461 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testNullMetaBlocks Thread=13, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3086 2023-03-27 09:00:45,480 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testNullMetaBlocks Thread=13 (was 13), OpenFileDescriptor=265 (was 263) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3086 (was 3086) 2023-03-27 09:00:45,490 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=13, OpenFileDescriptor=265, MaxFileDescriptor=60000, SystemLoadAverage=980, ProcessCount=178, AvailableMemoryMB=3086 2023-03-27 09:00:45,506 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2023-03-27 09:00:45,530 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on d68da7d10bd64ccbbafd834fa01b1122 2023-03-27 09:00:45,551 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-27 09:00:45,557 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2023-03-27 09:00:45,568 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2023-03-27 09:00:45,569 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2023-03-27 09:00:45,574 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2023-03-27 09:00:45,579 INFO [Time-limited test] bucket.BucketCache(1318): Shutdown bucket cache: IO persistent=false; path to write=null 2023-03-27 09:00:45,584 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2023-03-27 09:00:45,585 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2023-03-27 09:00:45,590 ERROR [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(975): WriterThread encountered error java.lang.IllegalArgumentException: ReferenceCount : 0 (expected: > 0) at org.apache.hbase.thirdparty.io.netty.util.internal.ObjectUtil.checkPositive(ObjectUtil.java:100) at org.apache.hadoop.hbase.nio.ByteBuff.checkRefCount(ByteBuff.java:77) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:683) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:43) at org.apache.hadoop.hbase.io.hfile.HFileBlock.getBufferReadOnly(HFileBlock.java:500) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1503) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1059) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:973) 2023-03-27 09:00:45,593 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(981): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2023-03-27 09:00:45,602 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=14 (was 13) Potentially hanging thread: Time-limited test-BucketCacheWriter-0 java.lang.Thread.interrupt(Thread.java:924) sun.misc.Resource.getBytes(Resource.java:149) sun.misc.URLClassPath$JarLoader$2.getBytes(URLClassPath.java:1003) java.net.URLClassLoader.defineClass(URLClassLoader.java:468) java.net.URLClassLoader.access$100(URLClassLoader.java:74) java.net.URLClassLoader$1.run(URLClassLoader.java:369) java.net.URLClassLoader$1.run(URLClassLoader.java:363) java.security.AccessController.doPrivileged(Native Method) java.net.URLClassLoader.findClass(URLClassLoader.java:362) java.lang.ClassLoader.loadClass(ClassLoader.java:418) sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) java.lang.ClassLoader.loadClass(ClassLoader.java:351) org.apache.log4j.spi.LoggingEvent.(LoggingEvent.java:177) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.error(Reload4jLoggerAdapter.java:555) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:975) - Thread LEAK? -, OpenFileDescriptor=265 (was 265), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=980 (was 980), ProcessCount=178 (was 178), AvailableMemoryMB=3054 (was 3086)