2024-01-06 23:56:22,496 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.io.hfile.TestHFile timeout: 13 mins 2024-01-06 23:56:22,790 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/64acc8cb-2f3b-bbd9-df0d-72a6018cfaa8 2024-01-06 23:56:22,951 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-01-06 23:56:23,448 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2024-01-06 23:56:23,482 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testEmptyHFile Thread=11, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=207, AvailableMemoryMB=869 2024-01-06 23:56:23,514 DEBUG [Time-limited test] util.ClassSize(228): Using Unsafe to estimate memory layout 2024-01-06 23:56:23,655 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-06 23:56:23,808 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-06 23:56:23,864 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testEmptyHFile 2024-01-06 23:56:24,170 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testEmptyHFile Thread=13 (was 11) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=291 (was 290) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=210 (was 207) - ProcessCount LEAK? -, AvailableMemoryMB=769 (was 869) 2024-01-06 23:56:24,183 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=13, OpenFileDescriptor=291, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=210, AvailableMemoryMB=769 2024-01-06 23:56:24,206 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-06 23:56:24,236 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on f98b9c1d276e4f97b49a912be6271641 2024-01-06 23:56:24,376 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=13 (was 13), OpenFileDescriptor=292 (was 291) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=207 (was 210), AvailableMemoryMB=757 (was 769) 2024-01-06 23:56:24,393 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=207, AvailableMemoryMB=757 2024-01-06 23:56:24,413 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-06 23:56:24,438 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on eee794c5d4754f88b05752bfab47be88 2024-01-06 23:56:24,481 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-06 23:56:24,498 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-06 23:56:24,512 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-06 23:56:24,522 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-06 23:56:24,524 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-06 23:56:24,534 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key eee794c5d4754f88b05752bfab47be88_116229 had block type DATA, but was found in L1 cache. 2024-01-06 23:56:24,542 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-06 23:56:24,542 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-06 23:56:24,543 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-06 23:56:24,552 ERROR [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(974): WriterThread encountered error java.lang.IllegalArgumentException: ReferenceCount : 0 (expected: > 0) at org.apache.hbase.thirdparty.io.netty.util.internal.ObjectUtil.checkPositive(ObjectUtil.java:100) at org.apache.hadoop.hbase.nio.ByteBuff.checkRefCount(ByteBuff.java:77) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:683) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:43) at org.apache.hadoop.hbase.io.hfile.HFileBlock.getBufferReadOnly(HFileBlock.java:506) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1504) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1059) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:972) 2024-01-06 23:56:24,554 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-06 23:56:24,565 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=14 (was 13) Potentially hanging thread: Time-limited test-BucketCacheWriter-2 java.lang.invoke.InnerClassLambdaMetafactory$ForwardingMethodGenerator.generate(InnerClassLambdaMetafactory.java:462) java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:289) java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:194) java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:304) java.lang.invoke.LambdaForm$DMH/1289479439.invokeStatic_L6_L(LambdaForm$DMH) java.lang.invoke.LambdaForm$BMH/1873653341.reinvoke(LambdaForm$BMH) java.lang.invoke.LambdaForm$MH/1531333864.invoke_MT(LambdaForm$MH) java.lang.invoke.CallSite.makeSite(CallSite.java:302) java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307) java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1058) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:972) - Thread LEAK? -, OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=207 (was 207), AvailableMemoryMB=714 (was 757) 2024-01-06 23:56:24,577 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=207, AvailableMemoryMB=713 2024-01-06 23:56:25,289 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 207), AvailableMemoryMB=693 (was 713) 2024-01-06 23:56:25,315 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=693 2024-01-06 23:56:25,323 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testCorruptTruncatedHFile 2024-01-06 23:56:25,364 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=690 (was 693) 2024-01-06 23:56:25,379 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=690 2024-01-06 23:56:25,397 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-06 23:56:25,404 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 7333c7123c454d7f849b4c44d84f477f 2024-01-06 23:56:25,431 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-06 23:56:25,437 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-06 23:56:25,450 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-06 23:56:25,450 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-06 23:56:25,457 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-06 23:56:25,463 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 7333c7123c454d7f849b4c44d84f477f_115884 had block type DATA, but was found in L1 cache. 2024-01-06 23:56:25,468 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-06 23:56:25,468 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-06 23:56:25,469 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-06 23:56:25,471 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-06 23:56:25,482 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=654 (was 690) 2024-01-06 23:56:25,495 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=13, OpenFileDescriptor=294, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=654 2024-01-06 23:56:25,509 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=13 (was 13), OpenFileDescriptor=293 (was 294), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=653 (was 654) 2024-01-06 23:56:25,520 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=652 2024-01-06 23:56:25,532 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-06 23:56:25,542 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on e3b7e0cd6ff546649dd8960d76a3223f 2024-01-06 23:56:25,575 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=15 (was 13) Potentially hanging thread: LruBlockCacheStatsExecutor Potentially hanging thread: Time-limited test.LruBlockCache.EvictionThread java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:902) - Thread LEAK? -, OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=652 (was 652) 2024-01-06 23:56:25,587 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testMetaBlocks Thread=13, OpenFileDescriptor=295, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=652 2024-01-06 23:56:25,664 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testMetaBlocks Thread=13 (was 13), OpenFileDescriptor=293 (was 295), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=652 (was 652) 2024-01-06 23:56:25,681 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testGetShortMidpoint Thread=13, OpenFileDescriptor=295, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=650 2024-01-06 23:56:25,708 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testGetShortMidpoint Thread=13 (was 13), OpenFileDescriptor=293 (was 295), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=649 (was 650) 2024-01-06 23:56:25,727 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCompressionOrdinance Thread=13, OpenFileDescriptor=295, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=649 2024-01-06 23:56:25,740 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCompressionOrdinance Thread=13 (was 13), OpenFileDescriptor=295 (was 295), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=649 (was 649) 2024-01-06 23:56:25,749 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=649 2024-01-06 23:56:25,763 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-06 23:56:25,769 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 5a1a0d9ee8b647758b3b0b279beb49df 2024-01-06 23:56:25,795 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-06 23:56:25,842 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-06 23:56:25,863 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-06 23:56:25,863 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-06 23:56:25,865 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-06 23:56:25,878 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 5a1a0d9ee8b647758b3b0b279beb49df_116161 had block type DATA, but was found in L1 cache. 2024-01-06 23:56:25,882 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-06 23:56:25,882 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-06 23:56:25,882 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-06 23:56:25,882 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-06 23:56:25,892 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=14 (was 13) Potentially hanging thread: ForkJoinPool.commonPool-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) - Thread LEAK? -, OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=615 (was 649) 2024-01-06 23:56:25,900 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testTFileFeatures Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=615 2024-01-06 23:56:25,904 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@319b0bbd, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-06 23:56:25,924 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@553a416f, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-06 23:56:25,957 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@38e4b42, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-06 23:56:25,974 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@2eb00001, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-06 23:56:25,998 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testTFileFeatures Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=614 (was 615) 2024-01-06 23:56:26,009 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testDBEShipped Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=614 2024-01-06 23:56:26,022 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_PREFIX 2024-01-06 23:56:26,032 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_DIFF 2024-01-06 23:56:26,042 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_FAST_DIFF 2024-01-06 23:56:26,051 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_ROW_INDEX_V1 2024-01-06 23:56:26,064 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testDBEShipped Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=612 (was 614) 2024-01-06 23:56:26,073 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testShortMidpointSameQual Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=612 2024-01-06 23:56:26,081 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testShortMidpointSameQual Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=612 (was 612) 2024-01-06 23:56:26,093 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testNullMetaBlocks Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=612 2024-01-06 23:56:26,116 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testNullMetaBlocks Thread=14 (was 14), OpenFileDescriptor=295 (was 293) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=188 (was 188), AvailableMemoryMB=611 (was 612) 2024-01-06 23:56:26,127 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=14, OpenFileDescriptor=295, MaxFileDescriptor=60000, SystemLoadAverage=749, ProcessCount=188, AvailableMemoryMB=610 2024-01-06 23:56:26,139 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-06 23:56:26,145 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 90330b3476454a37b84dad920e45c5fa 2024-01-06 23:56:26,170 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-06 23:56:26,172 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-06 23:56:26,181 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-06 23:56:26,182 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-06 23:56:26,183 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-06 23:56:26,185 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 90330b3476454a37b84dad920e45c5fa_115259 had block type DATA, but was found in L1 cache. 2024-01-06 23:56:26,186 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-06 23:56:26,188 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-06 23:56:26,188 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-06 23:56:26,190 ERROR [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(974): WriterThread encountered error java.lang.IllegalArgumentException: ReferenceCount : 0 (expected: > 0) at org.apache.hbase.thirdparty.io.netty.util.internal.ObjectUtil.checkPositive(ObjectUtil.java:100) at org.apache.hadoop.hbase.nio.ByteBuff.checkRefCount(ByteBuff.java:77) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:683) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:43) at org.apache.hadoop.hbase.io.hfile.HFileBlock.getBufferReadOnly(HFileBlock.java:506) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1504) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1059) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:972) 2024-01-06 23:56:26,191 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-06 23:56:26,207 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=19 (was 14) Potentially hanging thread: Time-limited test-BucketCacheWriter-1 org.apache.log4j.Category.callAppenders(Category.java:205) org.apache.log4j.Category.forcedLog(Category.java:384) org.apache.log4j.Category.log(Category.java:810) org.slf4j.impl.Reload4jLoggerAdapter.info(Reload4jLoggerAdapter.java:284) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:980) Potentially hanging thread: Time-limited test.LruBlockCache.EvictionThread java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:902) Potentially hanging thread: LruBlockCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Time-limited test-BucketCacheWriter-0 org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$$Lambda$48/1905587670.apply(Unknown Source) org.apache.hadoop.hbase.io.hfile.bucket.BucketEntry.(BucketEntry.java:110) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1499) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1059) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:972) Potentially hanging thread: BucketCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=295 (was 295), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=749 (was 749), ProcessCount=191 (was 188) - ProcessCount LEAK? -, AvailableMemoryMB=571 (was 610)