2024-01-26 04:55:25,807 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.io.hfile.TestHFile timeout: 13 mins 2024-01-26 04:55:26,044 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/68580a63-8ba0-3d8b-e71a-ee7badda0d65 2024-01-26 04:55:26,172 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2024-01-26 04:55:26,560 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2024-01-26 04:55:26,584 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testEmptyHFile Thread=11, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5564 2024-01-26 04:55:26,609 DEBUG [Time-limited test] util.ClassSize(228): Using Unsafe to estimate memory layout 2024-01-26 04:55:26,709 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2024-01-26 04:55:26,793 INFO [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2024-01-26 04:55:26,833 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testEmptyHFile 2024-01-26 04:55:27,043 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testEmptyHFile Thread=13 (was 11) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=291 (was 290) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5461 (was 5564) 2024-01-26 04:55:27,051 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=13, OpenFileDescriptor=291, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5461 2024-01-26 04:55:27,073 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-26 04:55:27,093 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on af0ead76624541d8858cfb41101d0af8 2024-01-26 04:55:27,179 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithoutBlockCache Thread=13 (was 13), OpenFileDescriptor=292 (was 291) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5449 (was 5461) 2024-01-26 04:55:27,187 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5448 2024-01-26 04:55:27,200 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-26 04:55:27,213 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 09e7cebc33fa4a8389514c1bbf023e0c 2024-01-26 04:55:27,247 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-26 04:55:27,265 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-26 04:55:27,277 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-26 04:55:27,289 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-26 04:55:27,290 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-26 04:55:27,299 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 09e7cebc33fa4a8389514c1bbf023e0c_115784 had block type DATA, but was found in L1 cache. 2024-01-26 04:55:27,304 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-26 04:55:27,305 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-26 04:55:27,307 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-26 04:55:27,316 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-26 04:55:27,323 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithAdaptiveLruCombinedBlockCache Thread=14 (was 13) Potentially hanging thread: Time-limited test-BucketCacheWriter-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getRAMQueueEntries(BucketCache.java:1154) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:966) - Thread LEAK? -, OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5382 (was 5448) 2024-01-26 04:55:27,333 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5381 2024-01-26 04:55:27,883 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptOutOfOrderHFileWrite Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5343 (was 5381) 2024-01-26 04:55:27,892 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5342 2024-01-26 04:55:27,900 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testCorruptTruncatedHFile 2024-01-26 04:55:27,930 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorruptTruncatedHFile Thread=13 (was 13), OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5342 (was 5342) 2024-01-26 04:55:27,940 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5342 2024-01-26 04:55:27,957 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-26 04:55:27,963 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on a7bdd11f0fa64696bf3be28207debd6f 2024-01-26 04:55:27,984 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-26 04:55:27,988 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-26 04:55:27,998 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-26 04:55:27,999 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-26 04:55:28,005 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-26 04:55:28,008 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key a7bdd11f0fa64696bf3be28207debd6f_116032 had block type DATA, but was found in L1 cache. 2024-01-26 04:55:28,009 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-26 04:55:28,009 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-26 04:55:28,009 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-26 04:55:28,013 ERROR [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(974): WriterThread encountered error java.lang.IllegalArgumentException: ReferenceCount : 0 (expected: > 0) at org.apache.hbase.thirdparty.io.netty.util.internal.ObjectUtil.checkPositive(ObjectUtil.java:100) at org.apache.hadoop.hbase.nio.ByteBuff.checkRefCount(ByteBuff.java:77) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:683) at org.apache.hadoop.hbase.nio.MultiByteBuff.duplicate(MultiByteBuff.java:43) at org.apache.hadoop.hbase.io.hfile.HFileBlock.getBufferReadOnly(HFileBlock.java:506) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1504) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1059) at org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:972) 2024-01-26 04:55:28,015 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-26 04:55:28,018 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithCombinedBlockCache Thread=15 (was 13) Potentially hanging thread: Time-limited test-BucketCacheWriter-0 jdk.internal.org.objectweb.asm.ByteVector.(ByteVector.java:95) jdk.internal.org.objectweb.asm.ClassWriter.toByteArray(ClassWriter.java:932) java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCodeBytes(InvokerBytecodeGenerator.java:727) java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCode(InvokerBytecodeGenerator.java:618) java.lang.invoke.LambdaForm.compileToBytecode(LambdaForm.java:654) java.lang.invoke.Invokers.invokeHandleForm(Invokers.java:289) java.lang.invoke.Invokers.methodHandleInvokeLinkerMethod(Invokers.java:198) java.lang.invoke.MethodHandleNatives.linkMethodImpl(MethodHandleNatives.java:431) java.lang.invoke.MethodHandleNatives.linkMethod(MethodHandleNatives.java:423) java.lang.invoke.CallSite.makeSite(CallSite.java:310) java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:307) java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297) java.util.Comparator.comparingLong(Comparator.java:510) org.apache.hadoop.hbase.io.hfile.bucket.BucketEntry.(BucketEntry.java:49) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$RAMQueueEntry.writeToCache(BucketCache.java:1498) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.doDrain(BucketCache.java:1059) org.apache.hadoop.hbase.io.hfile.bucket.BucketCache$WriterThread.run(BucketCache.java:972) Potentially hanging thread: BucketCacheStatsExecutor sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=292 (was 292), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5305 (was 5342) 2024-01-26 04:55:28,028 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=13, OpenFileDescriptor=292, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5305 2024-01-26 04:55:28,039 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCorrupt0LengthHFile Thread=13 (was 13), OpenFileDescriptor=293 (was 292) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5305 (was 5305) 2024-01-26 04:55:28,046 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5305 2024-01-26 04:55:28,062 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-26 04:55:28,069 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 5e976a75cb7e4c389f76731ca64e2dbd 2024-01-26 04:55:28,097 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLRUBlockCache Thread=13 (was 13), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5304 (was 5305) 2024-01-26 04:55:28,106 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testMetaBlocks Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5301 2024-01-26 04:55:28,158 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testMetaBlocks Thread=13 (was 13), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5300 (was 5301) 2024-01-26 04:55:28,167 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testGetShortMidpoint Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5300 2024-01-26 04:55:28,181 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testGetShortMidpoint Thread=13 (was 13), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5300 (was 5300) 2024-01-26 04:55:28,190 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testCompressionOrdinance Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5300 2024-01-26 04:55:28,200 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testCompressionOrdinance Thread=13 (was 13), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5300 (was 5300) 2024-01-26 04:55:28,206 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=13, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5300 2024-01-26 04:55:28,218 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-26 04:55:28,222 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 0a51ac69842a4381aa6da6db89902860 2024-01-26 04:55:28,238 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-26 04:55:28,271 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-26 04:55:28,279 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-26 04:55:28,279 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-26 04:55:28,280 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-26 04:55:28,289 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 0a51ac69842a4381aa6da6db89902860_115713 had block type DATA, but was found in L1 cache. 2024-01-26 04:55:28,296 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-26 04:55:28,296 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-26 04:55:28,296 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-26 04:55:28,296 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-26 04:55:28,304 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithTinyLfuCombinedBlockCache Thread=14 (was 13) Potentially hanging thread: ForkJoinPool.commonPool-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) - Thread LEAK? -, OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5260 (was 5300) 2024-01-26 04:55:28,311 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testTFileFeatures Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5260 2024-01-26 04:55:28,314 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@4a3030f2, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-26 04:55:28,333 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@167ce9cf, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-26 04:55:28,353 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@377ae5ba, compression=none cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-26 04:55:28,368 INFO [Time-limited test] hfile.TestHFile(498): writer=null, name=org.apache.hadoop.fs.FSDataOutputStream@714440bb, compression=gz cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 0 2024-01-26 04:55:28,386 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testTFileFeatures Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5254 (was 5260) 2024-01-26 04:55:28,393 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testDBEShipped Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5254 2024-01-26 04:55:28,398 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_PREFIX 2024-01-26 04:55:28,403 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_DIFF 2024-01-26 04:55:28,407 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_FAST_DIFF 2024-01-26 04:55:28,411 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on testDBEShipped_ROW_INDEX_V1 2024-01-26 04:55:28,422 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testDBEShipped Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5250 (was 5254) 2024-01-26 04:55:28,431 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testShortMidpointSameQual Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5250 2024-01-26 04:55:28,438 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testShortMidpointSameQual Thread=14 (was 14), OpenFileDescriptor=293 (was 293), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5251 (was 5250) - AvailableMemoryMB LEAK? - 2024-01-26 04:55:28,446 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testNullMetaBlocks Thread=14, OpenFileDescriptor=293, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5250 2024-01-26 04:55:28,466 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testNullMetaBlocks Thread=14 (was 14), OpenFileDescriptor=295 (was 293) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5248 (was 5250) 2024-01-26 04:55:28,473 INFO [Time-limited test] hbase.ResourceChecker(147): before: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=14, OpenFileDescriptor=295, MaxFileDescriptor=60000, SystemLoadAverage=241, ProcessCount=167, AvailableMemoryMB=5248 2024-01-26 04:55:28,490 WARN [Time-limited test] io.ByteBuffAllocator(181): The config keys hbase.ipc.server.reservoir.initial.buffer.size and hbase.ipc.server.reservoir.initial.max are deprecated now, instead please use hbase.server.allocator.buffer.size and hbase.server.allocator.max.buffer.count. In future release we will remove the two deprecated configs. 2024-01-26 04:55:28,495 DEBUG [Time-limited test] hfile.HFile$WriterFactory(307): Unable to set drop behind on 21b017b650d84a5387dc2957cbc7bb6a 2024-01-26 04:55:28,519 INFO [Time-limited test] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2024-01-26 04:55:28,520 INFO [Time-limited test] util.ByteBufferArray(64): Allocating buffers total=32 MB, sizePerBuffer=2 MB, count=16 2024-01-26 04:55:28,526 INFO [Time-limited test] bucket.BucketCache(278): Instantiating BucketCache with acceptableFactor: 0.95, minFactor: 0.85, extraFreeFactor: 0.1, singleFactor: 0.25, multiFactor: 0.5, memoryFactor: 0.25 2024-01-26 04:55:28,526 INFO [Time-limited test] bucket.BucketAllocator(339): Cache totalSize=31518720, buckets=15, bucket capacity=2101248=(4*525312)=(FEWEST_ITEMS_IN_BUCKET*(largest configured bucketcache size)) 2024-01-26 04:55:28,527 INFO [Time-limited test] bucket.BucketCache(320): Started bucket cache; ioengine=offheap, capacity=32 MB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2024-01-26 04:55:28,529 WARN [Time-limited test] hfile.CombinedBlockCache(106): Cache key 21b017b650d84a5387dc2957cbc7bb6a_115233 had block type DATA, but was found in L1 cache. 2024-01-26 04:55:28,530 INFO [Time-limited test-BucketCacheWriter-1] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-1 exiting, cacheEnabled=false 2024-01-26 04:55:28,530 INFO [Time-limited test-BucketCacheWriter-0] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-0 exiting, cacheEnabled=false 2024-01-26 04:55:28,530 INFO [Time-limited test] bucket.BucketCache(1319): Shutdown bucket cache: IO persistent=false; path to write=null 2024-01-26 04:55:28,540 INFO [Time-limited test-BucketCacheWriter-2] bucket.BucketCache$WriterThread(980): Time-limited test-BucketCacheWriter-2 exiting, cacheEnabled=false 2024-01-26 04:55:28,555 INFO [Time-limited test] hbase.ResourceChecker(175): after: io.hfile.TestHFile#testReaderWithLruCombinedBlockCache Thread=14 (was 14), OpenFileDescriptor=295 (was 295), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=241 (was 241), ProcessCount=167 (was 167), AvailableMemoryMB=5208 (was 5248)