2023-10-29 10:56:12,481 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa
2023-10-29 10:56:12,506 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStreamFSHLog timeout: 13 mins
2023-10-29 10:56:12,518 INFO  [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/cluster_8c197d25-4fbf-3f92-e73f-8830694ec9c2, deleteOnExit=true
2023-10-29 10:56:12,518 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/test.cache.data in system properties and HBase conf
2023-10-29 10:56:12,519 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/hadoop.tmp.dir in system properties and HBase conf
2023-10-29 10:56:12,519 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/hadoop.log.dir in system properties and HBase conf
2023-10-29 10:56:12,520 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/mapreduce.cluster.local.dir in system properties and HBase conf
2023-10-29 10:56:12,520 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/mapreduce.cluster.temp.dir in system properties and HBase conf
2023-10-29 10:56:12,521 INFO  [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF
2023-10-29 10:56:12,640 WARN  [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-10-29 10:56:13,024 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering
2023-10-29 10:56:13,029 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/yarn.node-labels.fs-store.root-dir in system properties and HBase conf
2023-10-29 10:56:13,029 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf
2023-10-29 10:56:13,030 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/yarn.nodemanager.log-dirs in system properties and HBase conf
2023-10-29 10:56:13,030 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf
2023-10-29 10:56:13,030 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf
2023-10-29 10:56:13,031 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf
2023-10-29 10:56:13,031 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf
2023-10-29 10:56:13,031 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/dfs.journalnode.edits.dir in system properties and HBase conf
2023-10-29 10:56:13,032 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf
2023-10-29 10:56:13,032 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/nfs.dump.dir in system properties and HBase conf
2023-10-29 10:56:13,033 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/java.io.tmpdir in system properties and HBase conf
2023-10-29 10:56:13,033 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/dfs.journalnode.edits.dir in system properties and HBase conf
2023-10-29 10:56:13,034 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf
2023-10-29 10:56:13,034 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/fs.s3a.committer.staging.tmp.path in system properties and HBase conf
Formatting using clusterid: testClusterID
2023-10-29 10:56:13,566 WARN  [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS
2023-10-29 10:56:13,571 WARN  [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS
2023-10-29 10:56:13,868 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2023-10-29 10:56:14,029 INFO  [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2023-10-29 10:56:14,043 WARN  [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j
2023-10-29 10:56:14,078 INFO  [Time-limited test] log.Slf4jLog(67): jetty-6.1.26
2023-10-29 10:56:14,110 INFO  [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/java.io.tmpdir/Jetty_localhost_42767_hdfs____.wsk6l8/webapp
2023-10-29 10:56:14,239 INFO  [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42767
2023-10-29 10:56:14,249 WARN  [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS
2023-10-29 10:56:14,249 WARN  [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS
2023-10-29 10:56:14,719 WARN  [Listener at localhost/35787] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j
2023-10-29 10:56:14,789 WARN  [Listener at localhost/35787] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS
2023-10-29 10:56:14,813 WARN  [Listener at localhost/35787] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j
2023-10-29 10:56:14,819 INFO  [Listener at localhost/35787] log.Slf4jLog(67): jetty-6.1.26
2023-10-29 10:56:14,823 INFO  [Listener at localhost/35787] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/java.io.tmpdir/Jetty_localhost_33123_datanode____u41pkp/webapp
2023-10-29 10:56:14,925 INFO  [Listener at localhost/35787] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33123
2023-10-29 10:56:15,208 WARN  [Listener at localhost/35411] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j
2023-10-29 10:56:15,225 WARN  [Listener at localhost/35411] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS
2023-10-29 10:56:15,229 WARN  [Listener at localhost/35411] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j
2023-10-29 10:56:15,231 INFO  [Listener at localhost/35411] log.Slf4jLog(67): jetty-6.1.26
2023-10-29 10:56:15,240 INFO  [Listener at localhost/35411] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/java.io.tmpdir/Jetty_localhost_36895_datanode____.uzhwnc/webapp
2023-10-29 10:56:15,337 INFO  [Listener at localhost/35411] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36895
2023-10-29 10:56:15,349 WARN  [Listener at localhost/42977] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j
2023-10-29 10:56:15,368 WARN  [Listener at localhost/42977] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS
2023-10-29 10:56:15,373 WARN  [Listener at localhost/42977] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j
2023-10-29 10:56:15,375 INFO  [Listener at localhost/42977] log.Slf4jLog(67): jetty-6.1.26
2023-10-29 10:56:15,381 INFO  [Listener at localhost/42977] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/java.io.tmpdir/Jetty_localhost_44987_datanode____vvbwpr/webapp
2023-10-29 10:56:15,488 INFO  [Listener at localhost/42977] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44987
2023-10-29 10:56:15,498 WARN  [Listener at localhost/38913] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j
2023-10-29 10:56:15,683 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3521005e891846: Processing first storage report for DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd from datanode a1ce4ba6-07a2-4753-8f74-c23555598f8d
2023-10-29 10:56:15,685 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3521005e891846: from storage DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd node DatanodeRegistration(127.0.0.1:43565, datanodeUuid=a1ce4ba6-07a2-4753-8f74-c23555598f8d, infoPort=40901, infoSecurePort=0, ipcPort=38913, storageInfo=lv=-57;cid=testClusterID;nsid=79868307;c=1698576973645), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0
2023-10-29 10:56:15,686 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3d11a28b4a1bea8: Processing first storage report for DS-9cc67798-dff7-442c-b368-0dae6e957a69 from datanode 9293b22f-87e1-47be-be13-eb912fda24ec
2023-10-29 10:56:15,686 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3d11a28b4a1bea8: from storage DS-9cc67798-dff7-442c-b368-0dae6e957a69 node DatanodeRegistration(127.0.0.1:36281, datanodeUuid=9293b22f-87e1-47be-be13-eb912fda24ec, infoPort=35419, infoSecurePort=0, ipcPort=35411, storageInfo=lv=-57;cid=testClusterID;nsid=79868307;c=1698576973645), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
2023-10-29 10:56:15,686 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe281af4df664cde1: Processing first storage report for DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79 from datanode 5c33fe30-3aa6-46c2-8deb-6449bdf0c2d6
2023-10-29 10:56:15,686 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe281af4df664cde1: from storage DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79 node DatanodeRegistration(127.0.0.1:45935, datanodeUuid=5c33fe30-3aa6-46c2-8deb-6449bdf0c2d6, infoPort=44521, infoSecurePort=0, ipcPort=42977, storageInfo=lv=-57;cid=testClusterID;nsid=79868307;c=1698576973645), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
2023-10-29 10:56:15,686 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3521005e891846: Processing first storage report for DS-3615bbea-0534-43bc-8757-e42971681dfc from datanode a1ce4ba6-07a2-4753-8f74-c23555598f8d
2023-10-29 10:56:15,686 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3521005e891846: from storage DS-3615bbea-0534-43bc-8757-e42971681dfc node DatanodeRegistration(127.0.0.1:43565, datanodeUuid=a1ce4ba6-07a2-4753-8f74-c23555598f8d, infoPort=40901, infoSecurePort=0, ipcPort=38913, storageInfo=lv=-57;cid=testClusterID;nsid=79868307;c=1698576973645), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
2023-10-29 10:56:15,687 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3d11a28b4a1bea8: Processing first storage report for DS-afff1334-31ef-4c69-a477-3d2de2b8c442 from datanode 9293b22f-87e1-47be-be13-eb912fda24ec
2023-10-29 10:56:15,687 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3d11a28b4a1bea8: from storage DS-afff1334-31ef-4c69-a477-3d2de2b8c442 node DatanodeRegistration(127.0.0.1:36281, datanodeUuid=9293b22f-87e1-47be-be13-eb912fda24ec, infoPort=35419, infoSecurePort=0, ipcPort=35411, storageInfo=lv=-57;cid=testClusterID;nsid=79868307;c=1698576973645), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
2023-10-29 10:56:15,687 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe281af4df664cde1: Processing first storage report for DS-bf341049-a3bd-4140-9801-b1c047b1e3a4 from datanode 5c33fe30-3aa6-46c2-8deb-6449bdf0c2d6
2023-10-29 10:56:15,687 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe281af4df664cde1: from storage DS-bf341049-a3bd-4140-9801-b1c047b1e3a4 node DatanodeRegistration(127.0.0.1:45935, datanodeUuid=5c33fe30-3aa6-46c2-8deb-6449bdf0c2d6, infoPort=44521, infoSecurePort=0, ipcPort=42977, storageInfo=lv=-57;cid=testClusterID;nsid=79868307;c=1698576973645), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0
2023-10-29 10:56:15,910 DEBUG [Listener at localhost/38913] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa
2023-10-29 10:56:15,925 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testLogrollWhileStreaming Thread=139, OpenFileDescriptor=402, MaxFileDescriptor=60000, SystemLoadAverage=153, ProcessCount=167, AvailableMemoryMB=4482
2023-10-29 10:56:15,928 DEBUG [Listener at localhost/38913] util.ClassSize(228): Using Unsafe to estimate memory layout
2023-10-29 10:56:16,552 INFO  [Listener at localhost/38913] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-10-29 10:56:16,578 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:16,646 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testLogrollWhileStreaming, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testLogrollWhileStreaming, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:16,671 DEBUG [Listener at localhost/38913] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream
java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate()
	at java.lang.Class.getMethod(Class.java:1786)
	at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.<clinit>(CommonFSUtils.java:750)
	at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160)
	at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:307)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:70)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:881)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:574)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:515)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:160)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
	at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:295)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase.initWAL(WALEntryStreamTestBase.java:151)
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream.setUp(TestBasicWALEntryStream.java:80)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)
2023-10-29 10:56:16,902 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1141): Slow sync cost: 130 ms, current pipeline: []
2023-10-29 10:56:16,903 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1698576976669
2023-10-29 10:56:16,903 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK]]
2023-10-29 10:56:17,020 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:17,216 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1698576976669 with entries=3, filesize=437 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1698576977197
2023-10-29 10:56:17,217 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:17,218 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(716): hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1698576976669 is not closed yet, will try archiving it next time
2023-10-29 10:56:17,222 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:17,222 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:17,223 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 319
2023-10-29 10:56:17,324 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 319
2023-10-29 10:56:17,424 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 319
2023-10-29 10:56:17,525 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 319
2023-10-29 10:56:17,626 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 319
2023-10-29 10:56:17,726 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 319
2023-10-29 10:56:17,735 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:17,741 DEBUG [Listener at localhost/38913] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1698576976669
2023-10-29 10:56:17,751 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 201
2023-10-29 10:56:17,752 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testLogrollWhileStreaming
2023-10-29 10:56:17,781 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:17,781 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testLogrollWhileStreaming:(num 1698576977197)
2023-10-29 10:56:17,792 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testLogrollWhileStreaming Thread=144 (was 139)
Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:44832 [Waiting for operation #4]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: LeaseRenewer:jenkins@localhost:35787
	java.lang.Thread.sleep(Native Method)
	org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411)
	org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76)
	org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:39188 [Waiting for operation #5]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@b1a6c5e
	java.lang.Thread.sleep(Native Method)
	org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253)
	org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46)
	org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: HBase-Metrics2-1
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=445 (was 402) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=153 (was 153), ProcessCount=167 (was 167), AvailableMemoryMB=4266 (was 4482)
2023-10-29 10:56:17,800 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testNewEntriesWhileStreaming Thread=144, OpenFileDescriptor=445, MaxFileDescriptor=60000, SystemLoadAverage=153, ProcessCount=167, AvailableMemoryMB=4266
2023-10-29 10:56:17,802 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:17,808 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testNewEntriesWhileStreaming, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testNewEntriesWhileStreaming, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:17,829 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testNewEntriesWhileStreaming/testNewEntriesWhileStreaming.1698576977809
2023-10-29 10:56:17,829 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:17,843 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 201
2023-10-29 10:56:17,851 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 437
2023-10-29 10:56:17,851 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testNewEntriesWhileStreaming
2023-10-29 10:56:17,863 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:17,863 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testNewEntriesWhileStreaming:(num 1698576977809)
2023-10-29 10:56:17,872 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testNewEntriesWhileStreaming Thread=145 (was 144)
Potentially hanging thread: IPC Parameter Sending Thread #1
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:44832 [Waiting for operation #8]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=445 (was 445), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=153 (was 153), ProcessCount=167 (was 167), AvailableMemoryMB=4265 (was 4266)
2023-10-29 10:56:17,881 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWithPartialWALEntryFailingFilter Thread=145, OpenFileDescriptor=445, MaxFileDescriptor=60000, SystemLoadAverage=153, ProcessCount=167, AvailableMemoryMB=4265
2023-10-29 10:56:17,882 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:17,886 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderWithPartialWALEntryFailingFilter, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:17,901 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter.1698576977888
2023-10-29 10:56:17,901 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:18,054 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-10-29 10:56:18,068 WARN  [Thread-184] regionserver.ReplicationSourceWALReader(177): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$PartialWALEntryFailingWALEntryFilter.filter(TestBasicWALEntryStream.java:840)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:361)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:224)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
2023-10-29 10:56:18,099 WARN  [Thread-184] regionserver.ReplicationSourceWALReader(177): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$PartialWALEntryFailingWALEntryFilter.filter(TestBasicWALEntryStream.java:840)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:361)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:224)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
2023-10-29 10:56:18,139 WARN  [Thread-184] regionserver.ReplicationSourceWALReader(177): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$PartialWALEntryFailingWALEntryFilter.filter(TestBasicWALEntryStream.java:840)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:361)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:224)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
2023-10-29 10:56:18,190 DEBUG [Thread-184] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:18,191 DEBUG [Thread-184] regionserver.ReplicationSourceWALReader(162): Read 3 WAL entries eligible for replication
2023-10-29 10:56:18,191 DEBUG [Thread-184] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:18,208 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter
2023-10-29 10:56:18,212 DEBUG [Thread-184] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:18,220 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:18,221 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderWithPartialWALEntryFailingFilter:(num 1698576977888)
2023-10-29 10:56:18,222 WARN  [Thread-184] regionserver.WALEntryStream(207): Couldn't get file length information about log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter.1698576977888, it was closed cleanly currently replicating from: hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter.1698576977888 at position: 458

2023-10-29 10:56:18,223 DEBUG [Thread-184] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter.1698576977888
2023-10-29 10:56:18,223 DEBUG [Thread-184] regionserver.ReplicationSourceWALReader(162): Read 0 WAL entries eligible for replication
2023-10-29 10:56:18,230 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWithPartialWALEntryFailingFilter Thread=147 (was 145)
Potentially hanging thread: Thread-184
	java.lang.Object.wait(Native Method)
	java.lang.Object.wait(Object.java:502)
	org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
	org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1528)
	org.apache.hadoop.ipc.Client.call(Client.java:1486)
	org.apache.hadoop.ipc.Client.call(Client.java:1385)
	org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
	com.sun.proxy.$Proxy31.getFileInfo(Unknown Source)
	org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:800)
	sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	com.sun.proxy.$Proxy34.getFileInfo(Unknown Source)
	org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
	org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1523)
	org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520)
	org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkAllBytesParsed(WALEntryStream.java:205)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:183)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:102)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:258)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:53644 [Waiting for operation #9]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:39188 [Waiting for operation #10]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=447 (was 445) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=153 (was 153), ProcessCount=167 (was 167), AvailableMemoryMB=4263 (was 4265)
2023-10-29 10:56:18,240 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionForRecoveredQueueWithMultipleLogs Thread=147, OpenFileDescriptor=447, MaxFileDescriptor=60000, SystemLoadAverage=153, ProcessCount=167, AvailableMemoryMB=4263
2023-10-29 10:56:18,241 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:18,246 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testEOFExceptionForRecoveredQueueWithMultipleLogs, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionForRecoveredQueueWithMultipleLogs, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:18,264 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionForRecoveredQueueWithMultipleLogs/testEOFExceptionForRecoveredQueueWithMultipleLogs.1698576978248
2023-10-29 10:56:18,264 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:18,374 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=10
2023-10-29 10:56:18,385 DEBUG [Thread-207] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/user/jenkins/log.1
2023-10-29 10:56:18,389 WARN  [Thread-207] regionserver.ReplicationSourceWALReader(302): Forcing removal of 0 length log in queue: hdfs://localhost:35787/user/jenkins/log.2
2023-10-29 10:56:18,389 DEBUG [Thread-207] regionserver.ReplicationSourceWALReader(330): Read 3 WAL entries eligible for replication
2023-10-29 10:56:18,389 DEBUG [Thread-207] regionserver.ReplicationSourceWALReader(246): Stopping the replication source wal reader
2023-10-29 10:56:18,393 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionForRecoveredQueueWithMultipleLogs
2023-10-29 10:56:18,402 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:18,402 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testEOFExceptionForRecoveredQueueWithMultipleLogs:(num 1698576978248)
2023-10-29 10:56:18,412 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionForRecoveredQueueWithMultipleLogs Thread=147 (was 147), OpenFileDescriptor=453 (was 447) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 153) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=4262 (was 4263)
2023-10-29 10:56:18,420 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionInOldWALsDirectory Thread=147, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4262
2023-10-29 10:56:18,421 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:18,424 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testEOFExceptionInOldWALsDirectory, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:18,440 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978425
2023-10-29 10:56:18,440 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:18,453 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978425 with entries=0, filesize=83 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978440
2023-10-29 10:56:18,453 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:18,454 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(716): hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978425 is not closed yet, will try archiving it next time
2023-10-29 10:56:18,455 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:18,461 INFO  [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978425 to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testEOFExceptionInOldWALsDirectory.1698576978425
2023-10-29 10:56:18,556 INFO  [Listener at localhost/38913] wal.AbstractFSWALProvider(464): Log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978425 was moved to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testEOFExceptionInOldWALsDirectory.1698576978425
2023-10-29 10:56:18,568 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-10-29 10:56:18,569 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [10,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:18,572 INFO  [Thread-224] wal.AbstractFSWALProvider(464): Log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978425 was moved to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testEOFExceptionInOldWALsDirectory.1698576978425
2023-10-29 10:56:18,575 INFO  [Thread-224] wal.AbstractFSWALProvider(464): Log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978425 was moved to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testEOFExceptionInOldWALsDirectory.1698576978425
2023-10-29 10:56:18,576 WARN  [Thread-224] regionserver.ReplicationSourceWALReader(302): Forcing removal of 0 length log in queue: hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testEOFExceptionInOldWALsDirectory.1698576978425
2023-10-29 10:56:18,583 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-10-29 10:56:18,600 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-10-29 10:56:18,617 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-10-29 10:56:18,633 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-10-29 10:56:18,650 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-10-29 10:56:18,782 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory
2023-10-29 10:56:18,793 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:18,793 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testEOFExceptionInOldWALsDirectory:(num 1698576978440)
2023-10-29 10:56:18,801 INFO  [Thread-224] wal.AbstractFSWALProvider(464): Log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1698576978440 was moved to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testEOFExceptionInOldWALsDirectory.1698576978440
2023-10-29 10:56:18,803 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionInOldWALsDirectory Thread=148 (was 147)
Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:53644 [Waiting for operation #13]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: Thread-224
	java.lang.Object.wait(Native Method)
	java.lang.Object.wait(Object.java:502)
	org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
	org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1528)
	org.apache.hadoop.ipc.Client.call(Client.java:1486)
	org.apache.hadoop.ipc.Client.call(Client.java:1385)
	org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
	com.sun.proxy.$Proxy31.getBlockLocations(Unknown Source)
	org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:263)
	sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	com.sun.proxy.$Proxy34.getBlockLocations(Unknown Source)
	org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:849)
	org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:838)
	org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:827)
	org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:332)
	org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:291)
	org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:276)
	org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1066)
	org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
	org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:323)
	org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
	org.apache.hadoop.fs.FileSystem.open(FileSystem.java:914)
	org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:214)
	org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.reset(ProtobufLogReader.java:160)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.resetReader(WALEntryStream.java:381)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.reset(WALEntryStream.java:159)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:149)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:39188 [Waiting for operation #16]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4240 (was 4262)
2023-10-29 10:56:18,811 DEBUG [Thread-224] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testEOFExceptionInOldWALsDirectory.1698576978440
2023-10-29 10:56:18,811 DEBUG [Thread-224] regionserver.ReplicationSourceWALReader(162): Read 0 WAL entries eligible for replication
2023-10-29 10:56:18,812 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderRecovered Thread=148, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4239
2023-10-29 10:56:18,814 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:18,816 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderRecovered, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderRecovered, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:18,827 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1698576978817
2023-10-29 10:56:18,827 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:18,842 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1698576978817 with entries=10, filesize=1.30 KB; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1698576978830
2023-10-29 10:56:18,842 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:18,842 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(716): hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1698576978817 is not closed yet, will try archiving it next time
2023-10-29 10:56:18,849 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderRecovered
2023-10-29 10:56:18,858 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=10, replicationBatchQueueCapacity=1
2023-10-29 10:56:18,866 DEBUG [Thread-246] regionserver.ReplicationSourceWALReader(162): Read 10 WAL entries eligible for replication
2023-10-29 10:56:18,869 DEBUG [Thread-246] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1698576978817
2023-10-29 10:56:18,874 DEBUG [Thread-246] regionserver.ReplicationSourceWALReader(162): Read 0 WAL entries eligible for replication
2023-10-29 10:56:18,879 DEBUG [Thread-246] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1698576978830
2023-10-29 10:56:18,879 DEBUG [Thread-246] regionserver.ReplicationSourceWALReader(162): Read 5 WAL entries eligible for replication
2023-10-29 10:56:18,880 DEBUG [Thread-246] regionserver.ReplicationSourceWALReader(246): Stopping the replication source wal reader
2023-10-29 10:56:18,884 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:18,884 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderRecovered:(num 1698576978830)
2023-10-29 10:56:18,893 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderRecovered Thread=148 (was 148), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4238 (was 4239)
2023-10-29 10:56:18,901 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionForRecoveredQueue Thread=148, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4237
2023-10-29 10:56:18,902 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:18,906 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testEOFExceptionForRecoveredQueue, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionForRecoveredQueue, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:18,918 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionForRecoveredQueue/testEOFExceptionForRecoveredQueue.1698576978906
2023-10-29 10:56:18,918 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK]]
2023-10-29 10:56:18,925 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=10
2023-10-29 10:56:18,930 WARN  [Thread-256] regionserver.ReplicationSourceWALReader(302): Forcing removal of 0 length log in queue: emptyLog
2023-10-29 10:56:18,930 DEBUG [Thread-256] regionserver.ReplicationSourceWALReader(246): Stopping the replication source wal reader
2023-10-29 10:56:18,931 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEOFExceptionForRecoveredQueue
2023-10-29 10:56:18,939 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:18,940 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testEOFExceptionForRecoveredQueue:(num 1698576978906)
2023-10-29 10:56:18,948 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionForRecoveredQueue Thread=148 (was 148), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4237 (was 4237)
2023-10-29 10:56:18,957 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWrongPosition Thread=148, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4237
2023-10-29 10:56:18,958 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:18,961 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderWrongPosition, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:18,972 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576978962
2023-10-29 10:56:18,972 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:18,990 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576978962 with entries=1, filesize=208 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576978974
2023-10-29 10:56:18,990 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:18,990 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(716): hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576978962 is not closed yet, will try archiving it next time
2023-10-29 10:56:18,996 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:18,999 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-10-29 10:56:19,008 DEBUG [Thread-273] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576978962
2023-10-29 10:56:19,015 DEBUG [Thread-273] regionserver.ReplicationSourceWALReader(162): Read 1 WAL entries eligible for replication
2023-10-29 10:56:19,022 DEBUG [Thread-273] wal.ProtobufLogReader(420): EOF at position 2583
2023-10-29 10:56:19,022 DEBUG [Thread-273] regionserver.ReplicationSourceWALReader(162): Read 20 WAL entries eligible for replication
2023-10-29 10:56:19,022 DEBUG [Thread-273] wal.ProtobufLogReader(420): EOF at position 2583
2023-10-29 10:56:19,034 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576978974 with entries=20, filesize=2.52 KB; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576979022
2023-10-29 10:56:19,034 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:19,034 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(716): hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576978974 is not closed yet, will try archiving it next time
2023-10-29 10:56:19,050 DEBUG [Thread-273] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576978974
2023-10-29 10:56:19,056 DEBUG [Thread-273] regionserver.ReplicationSourceWALReader(162): Read 0 WAL entries eligible for replication
2023-10-29 10:56:19,058 DEBUG [Thread-273] wal.ProtobufLogReader(420): EOF at position 1333
2023-10-29 10:56:19,058 DEBUG [Thread-273] regionserver.ReplicationSourceWALReader(162): Read 10 WAL entries eligible for replication
2023-10-29 10:56:19,058 DEBUG [Thread-273] wal.ProtobufLogReader(420): EOF at position 1333
2023-10-29 10:56:19,059 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition
2023-10-29 10:56:19,073 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 3 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:19,073 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderWrongPosition:(num 1698576979022)
2023-10-29 10:56:19,075 INFO  [Thread-273] wal.AbstractFSWALProvider(464): Log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1698576979022 was moved to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testReplicationSourceWALReaderWrongPosition.1698576979022
2023-10-29 10:56:19,079 DEBUG [Thread-273] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testReplicationSourceWALReaderWrongPosition.1698576979022
2023-10-29 10:56:19,079 DEBUG [Thread-273] regionserver.ReplicationSourceWALReader(162): Read 0 WAL entries eligible for replication
2023-10-29 10:56:19,083 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWrongPosition Thread=149 (was 148)
Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:44832 [Waiting for operation #22]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:53644 [Waiting for operation #16]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:39188 [Waiting for operation #20]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: Thread-273
	java.util.Formatter$FormatSpecifier.checkGeneral(Formatter.java:2955)
	java.util.Formatter$FormatSpecifier.<init>(Formatter.java:2725)
	java.util.Formatter.parse(Formatter.java:2560)
	java.util.Formatter.format(Formatter.java:2501)
	java.util.Formatter.format(Formatter.java:2455)
	java.lang.String.format(String.java:2940)
	org.apache.hadoop.fs.FileSystem.get(FileSystem.java:464)
	org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
	org.apache.hadoop.hbase.wal.AbstractFSWALProvider.findArchivedLog(AbstractFSWALProvider.java:459)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.resetReader(WALEntryStream.java:385)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:180)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:102)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:258)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:145)
 - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4235 (was 4237)
2023-10-29 10:56:19,092 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testWALKeySerialization Thread=149, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4235
2023-10-29 10:56:19,093 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:19,096 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testWALKeySerialization, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testWALKeySerialization, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:19,111 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testWALKeySerialization/testWALKeySerialization.1698576979097
2023-10-29 10:56:19,111 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:19,118 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testWALKeySerialization
2023-10-29 10:56:19,125 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:19,125 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testWALKeySerialization:(num 1698576979097)
2023-10-29 10:56:19,136 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testWALKeySerialization Thread=149 (was 149), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4234 (was 4235)
2023-10-29 10:56:19,145 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderDisabled Thread=149, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4234
2023-10-29 10:56:19,147 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:19,150 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderDisabled, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderDisabled, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:19,162 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderDisabled/testReplicationSourceWALReaderDisabled.1698576979151
2023-10-29 10:56:19,162 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:19,173 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-10-29 10:56:19,175 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:19,287 DEBUG [Thread-301] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:19,288 DEBUG [Thread-301] regionserver.ReplicationSourceWALReader(162): Read 3 WAL entries eligible for replication
2023-10-29 10:56:19,288 DEBUG [Thread-301] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:19,288 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderDisabled
2023-10-29 10:56:19,298 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:19,298 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderDisabled:(num 1698576979151)
2023-10-29 10:56:19,300 INFO  [Thread-301] wal.AbstractFSWALProvider(464): Log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderDisabled/testReplicationSourceWALReaderDisabled.1698576979151 was moved to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testReplicationSourceWALReaderDisabled.1698576979151
2023-10-29 10:56:19,307 DEBUG [Thread-301] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testReplicationSourceWALReaderDisabled.1698576979151
2023-10-29 10:56:19,307 DEBUG [Thread-301] regionserver.ReplicationSourceWALReader(162): Read 0 WAL entries eligible for replication
2023-10-29 10:56:19,315 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderDisabled Thread=151 (was 149)
Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:44832 [Waiting for operation #24]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: ForkJoinPool.commonPool-worker-1
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
	java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
	java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:53644 [Waiting for operation #18]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: Thread-301
	java.lang.Thread.sleep(Native Method)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.handleEmptyWALEntryBatch(ReplicationSourceWALReader.java:251)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:148)
 - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4234 (was 4234)
2023-10-29 10:56:19,324 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testCleanClosedWALs Thread=151, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4231
2023-10-29 10:56:19,327 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:19,330 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testCleanClosedWALs, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testCleanClosedWALs, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:19,340 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testCleanClosedWALs/testCleanClosedWALs.1698576979330
2023-10-29 10:56:19,340 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK]]
2023-10-29 10:56:19,342 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:19,360 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testCleanClosedWALs/testCleanClosedWALs.1698576979330 with entries=1, filesize=208 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testCleanClosedWALs/testCleanClosedWALs.1698576979350
2023-10-29 10:56:19,360 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:19,361 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(716): hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testCleanClosedWALs/testCleanClosedWALs.1698576979330 is not closed yet, will try archiving it next time
2023-10-29 10:56:19,362 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:19,362 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 208
2023-10-29 10:56:19,463 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 208
2023-10-29 10:56:19,467 DEBUG [Listener at localhost/38913] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testCleanClosedWALs/testCleanClosedWALs.1698576979330
2023-10-29 10:56:19,474 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testCleanClosedWALs
2023-10-29 10:56:19,887 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:19,887 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testCleanClosedWALs:(num 1698576979350)
2023-10-29 10:56:19,896 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testCleanClosedWALs Thread=151 (was 151), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4230 (was 4231)
2023-10-29 10:56:19,905 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReadBeyondCommittedLength Thread=151, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4230
2023-10-29 10:56:19,906 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:19,909 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReadBeyondCommittedLength, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReadBeyondCommittedLength, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:19,922 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReadBeyondCommittedLength/testReadBeyondCommittedLength.1698576979909
2023-10-29 10:56:19,922 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:19,932 DEBUG [Listener at localhost/38913] regionserver.WALEntryStream(266): The provider tells us the valid length for hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReadBeyondCommittedLength/testReadBeyondCommittedLength.1698576979909 is 318, but we have advanced to 319
2023-10-29 10:56:20,944 DEBUG [Listener at localhost/38913] regionserver.WALEntryStream(266): The provider tells us the valid length for hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReadBeyondCommittedLength/testReadBeyondCommittedLength.1698576979909 is 318, but we have advanced to 319
2023-10-29 10:56:20,956 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 319
2023-10-29 10:56:20,956 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReadBeyondCommittedLength
2023-10-29 10:56:20,963 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:20,964 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReadBeyondCommittedLength:(num 1698576979909)
2023-10-29 10:56:20,974 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReadBeyondCommittedLength Thread=151 (was 151), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4229 (was 4230)
2023-10-29 10:56:20,984 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testResumeStreamingFromPosition Thread=151, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4229
2023-10-29 10:56:20,985 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:20,988 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testResumeStreamingFromPosition, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testResumeStreamingFromPosition, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:20,999 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testResumeStreamingFromPosition/testResumeStreamingFromPosition.1698576980989
2023-10-29 10:56:20,999 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:21,018 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 437
2023-10-29 10:56:21,019 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testResumeStreamingFromPosition
2023-10-29 10:56:21,025 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:21,025 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testResumeStreamingFromPosition:(num 1698576980989)
2023-10-29 10:56:21,035 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testResumeStreamingFromPosition Thread=151 (was 151), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4229 (was 4229)
2023-10-29 10:56:21,044 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReader Thread=151, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4229
2023-10-29 10:56:21,045 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:21,048 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReader, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReader, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:21,057 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReader/testReplicationSourceWALReader.1698576981048
2023-10-29 10:56:21,057 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:21,069 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-10-29 10:56:21,078 DEBUG [Thread-353] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:21,078 DEBUG [Thread-353] regionserver.ReplicationSourceWALReader(162): Read 3 WAL entries eligible for replication
2023-10-29 10:56:21,078 DEBUG [Thread-353] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:21,095 DEBUG [Thread-353] wal.ProtobufLogReader(420): EOF at position 578
2023-10-29 10:56:21,095 DEBUG [Thread-353] regionserver.ReplicationSourceWALReader(162): Read 1 WAL entries eligible for replication
2023-10-29 10:56:21,095 DEBUG [Thread-353] wal.ProtobufLogReader(420): EOF at position 578
2023-10-29 10:56:21,096 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReader
2023-10-29 10:56:21,102 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:21,103 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReader:(num 1698576981048)
2023-10-29 10:56:21,107 INFO  [Thread-353] wal.AbstractFSWALProvider(464): Log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReader/testReplicationSourceWALReader.1698576981048 was moved to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testReplicationSourceWALReader.1698576981048
2023-10-29 10:56:21,112 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReader Thread=152 (was 151)
Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:53644 [Waiting for operation #29]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: Thread-353
	java.lang.Thread.sleep(Native Method)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.handleEmptyWALEntryBatch(ReplicationSourceWALReader.java:251)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:148)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-922009149_17 at /127.0.0.1:39188 [Waiting for operation #33]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4228 (was 4229)
2023-10-29 10:56:21,114 DEBUG [Thread-353] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testReplicationSourceWALReader.1698576981048
2023-10-29 10:56:21,114 DEBUG [Thread-353] regionserver.ReplicationSourceWALReader(162): Read 0 WAL entries eligible for replication
2023-10-29 10:56:21,121 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWithFailingFilter Thread=152, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4228
2023-10-29 10:56:21,122 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:21,126 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderWithFailingFilter, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithFailingFilter, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:21,135 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithFailingFilter/testReplicationSourceWALReaderWithFailingFilter.1698576981126
2023-10-29 10:56:21,135 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK]]
2023-10-29 10:56:21,144 INFO  [Listener at localhost/38913] regionserver.ReplicationSourceWALReader(119): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-10-29 10:56:21,151 WARN  [Thread-365] regionserver.ReplicationSourceWALReader(177): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:361)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:224)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
2023-10-29 10:56:21,178 WARN  [Thread-365] regionserver.ReplicationSourceWALReader(177): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:361)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:224)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
2023-10-29 10:56:21,215 WARN  [Thread-365] regionserver.ReplicationSourceWALReader(177): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:361)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:224)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
2023-10-29 10:56:21,262 WARN  [Thread-365] regionserver.ReplicationSourceWALReader(177): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:361)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:224)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
2023-10-29 10:56:21,319 WARN  [Thread-365] regionserver.ReplicationSourceWALReader(177): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:361)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:224)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:157)
2023-10-29 10:56:21,387 DEBUG [Thread-365] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:21,388 DEBUG [Thread-365] regionserver.ReplicationSourceWALReader(162): Read 3 WAL entries eligible for replication
2023-10-29 10:56:21,388 DEBUG [Thread-365] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:21,388 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithFailingFilter
2023-10-29 10:56:21,394 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:21,394 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderWithFailingFilter:(num 1698576981126)
2023-10-29 10:56:21,399 INFO  [Thread-365] wal.AbstractFSWALProvider(464): Log hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testReplicationSourceWALReaderWithFailingFilter/testReplicationSourceWALReaderWithFailingFilter.1698576981126 was moved to hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testReplicationSourceWALReaderWithFailingFilter.1698576981126
2023-10-29 10:56:21,404 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWithFailingFilter Thread=153 (was 152) - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4227 (was 4228)
2023-10-29 10:56:21,407 DEBUG [Thread-365] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs/testReplicationSourceWALReaderWithFailingFilter.1698576981126
2023-10-29 10:56:21,408 DEBUG [Thread-365] regionserver.ReplicationSourceWALReader(162): Read 0 WAL entries eligible for replication
2023-10-29 10:56:21,415 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEmptyStream Thread=153, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4227
2023-10-29 10:56:21,416 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:21,419 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testEmptyStream, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEmptyStream, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:21,428 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEmptyStream/testEmptyStream.1698576981419
2023-10-29 10:56:21,428 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:21,435 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 83
2023-10-29 10:56:21,437 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testEmptyStream
2023-10-29 10:56:21,443 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:21,443 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testEmptyStream:(num 1698576981419)
2023-10-29 10:56:21,452 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEmptyStream Thread=153 (was 153), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4227 (was 4227)
2023-10-29 10:56:21,461 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testAppendsWithRolls Thread=153, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4227
2023-10-29 10:56:21,462 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:21,465 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testAppendsWithRolls, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testAppendsWithRolls, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:21,474 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testAppendsWithRolls/testAppendsWithRolls.1698576981465
2023-10-29 10:56:21,474 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK]]
2023-10-29 10:56:21,482 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 208
2023-10-29 10:56:21,483 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 208
2023-10-29 10:56:21,483 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 208
2023-10-29 10:56:21,485 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:21,503 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testAppendsWithRolls/testAppendsWithRolls.1698576981465 with entries=3, filesize=458 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testAppendsWithRolls/testAppendsWithRolls.1698576981491
2023-10-29 10:56:21,504 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:21,504 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(716): hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testAppendsWithRolls/testAppendsWithRolls.1698576981465 is not closed yet, will try archiving it next time
2023-10-29 10:56:21,506 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:21,511 INFO  [Listener at localhost/38913] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-10-29 10:56:21,604 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(425): Encountered a malformed edit, seeking back to last good position in file, from 466 to 458
java.io.EOFException: Invalid PB, EOF? Ignoring; originalPosition=458, currentPosition=466
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:354)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:95)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:83)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:259)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:173)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:102)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.peek(WALEntryStream.java:111)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.next(WALEntryStream.java:118)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.access$001(WALEntryStreamTestBase.java:82)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.lambda$next$0(WALEntryStreamTestBase.java:95)
	at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183)
	at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.next(WALEntryStreamTestBase.java:94)
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream.testAppendsWithRolls(TestBasicWALEntryStream.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: encoded_region_name, table_name, log_sequence_number, write_time
	at org.apache.hbase.thirdparty.com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:68)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:232)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:237)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.parseDelimitedFrom(ProtobufUtil.java:3578)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:348)
	... 40 more
2023-10-29 10:56:21,608 DEBUG [Listener at localhost/38913] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testAppendsWithRolls/testAppendsWithRolls.1698576981465
2023-10-29 10:56:21,614 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 208
2023-10-29 10:56:21,614 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testAppendsWithRolls
2023-10-29 10:56:21,622 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:21,622 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testAppendsWithRolls:(num 1698576981491)
2023-10-29 10:56:21,631 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testAppendsWithRolls Thread=152 (was 153), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4225 (was 4227)
2023-10-29 10:56:21,641 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testPosition Thread=152, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4225
2023-10-29 10:56:21,642 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:21,645 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testPosition, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testPosition, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:21,656 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testPosition/testPosition.1698576981645
2023-10-29 10:56:21,657 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK]]
2023-10-29 10:56:21,670 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 458
2023-10-29 10:56:21,671 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testPosition
2023-10-29 10:56:21,676 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:21,676 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testPosition:(num 1698576981645)
2023-10-29 10:56:21,685 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testPosition Thread=152 (was 152), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4225 (was 4225)
2023-10-29 10:56:21,694 INFO  [Listener at localhost/38913] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testSizeOfLogQueue Thread=152, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=157, ProcessCount=167, AvailableMemoryMB=4225
2023-10-29 10:56:21,695 INFO  [Listener at localhost/38913] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-10-29 10:56:21,697 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testSizeOfLogQueue, suffix=, logDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testSizeOfLogQueue, archiveDir=hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs, maxLogs=32
2023-10-29 10:56:21,708 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1698576981698
2023-10-29 10:56:21,708 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK]]
2023-10-29 10:56:21,721 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1698576981698 with entries=1, filesize=208 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1698576981710
2023-10-29 10:56:21,722 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36281,DS-9cc67798-dff7-442c-b368-0dae6e957a69,DISK], DatanodeInfoWithStorage[127.0.0.1:43565,DS-f954f66e-5178-4d97-a3bf-a4a79f6c5efd,DISK], DatanodeInfoWithStorage[127.0.0.1:45935,DS-ccffe4ee-9247-47cc-97c4-c21a7d8dbd79,DISK]]
2023-10-29 10:56:21,722 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(716): hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1698576981698 is not closed yet, will try archiving it next time
2023-10-29 10:56:21,732 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(425): Encountered a malformed edit, seeking back to last good position in file, from 216 to 208
java.io.EOFException: Invalid PB, EOF? Ignoring; originalPosition=208, currentPosition=216
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:354)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:95)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:83)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:259)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:173)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:102)
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream.testSizeOfLogQueue(TestBasicWALEntryStream.java:707)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: encoded_region_name, table_name, log_sequence_number, write_time
	at org.apache.hbase.thirdparty.com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:68)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:232)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:237)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.parseDelimitedFrom(ProtobufUtil.java:3578)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:348)
	... 33 more
2023-10-29 10:56:21,736 DEBUG [Listener at localhost/38913] regionserver.WALEntryStream(248): EOF, closing hdfs://localhost:35787/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1698576981698
2023-10-29 10:56:21,741 DEBUG [Listener at localhost/38913] wal.ProtobufLogReader(420): EOF at position 83
2023-10-29 10:56:21,741 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/WALs/testSizeOfLogQueue
2023-10-29 10:56:21,751 DEBUG [Listener at localhost/38913] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/oldWALs
2023-10-29 10:56:21,751 INFO  [Listener at localhost/38913] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testSizeOfLogQueue:(num 1698576981710)
2023-10-29 10:56:21,761 INFO  [Listener at localhost/38913] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testSizeOfLogQueue Thread=152 (was 152), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=157 (was 157), ProcessCount=167 (was 167), AvailableMemoryMB=4224 (was 4225)
2023-10-29 10:56:21,761 INFO  [Listener at localhost/38913] hbase.HBaseTestingUtility(1286): Shutting down minicluster
2023-10-29 10:56:21,762 WARN  [Listener at localhost/38913] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called
2023-10-29 10:56:21,766 INFO  [Listener at localhost/38913] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2023-10-29 10:56:21,868 WARN  [BP-1345938757-172.31.14.131-1698576973645 heartbeating to localhost/127.0.0.1:35787] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted
2023-10-29 10:56:21,868 WARN  [BP-1345938757-172.31.14.131-1698576973645 heartbeating to localhost/127.0.0.1:35787] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1345938757-172.31.14.131-1698576973645 (Datanode Uuid a1ce4ba6-07a2-4753-8f74-c23555598f8d) service to localhost/127.0.0.1:35787
2023-10-29 10:56:21,870 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/cluster_8c197d25-4fbf-3f92-e73f-8830694ec9c2/dfs/data/data5/current/BP-1345938757-172.31.14.131-1698576973645] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-10-29 10:56:21,870 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/cluster_8c197d25-4fbf-3f92-e73f-8830694ec9c2/dfs/data/data6/current/BP-1345938757-172.31.14.131-1698576973645] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-10-29 10:56:21,871 WARN  [Listener at localhost/38913] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called
2023-10-29 10:56:21,874 INFO  [Listener at localhost/38913] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2023-10-29 10:56:21,980 WARN  [BP-1345938757-172.31.14.131-1698576973645 heartbeating to localhost/127.0.0.1:35787] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted
2023-10-29 10:56:21,980 WARN  [BP-1345938757-172.31.14.131-1698576973645 heartbeating to localhost/127.0.0.1:35787] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1345938757-172.31.14.131-1698576973645 (Datanode Uuid 5c33fe30-3aa6-46c2-8deb-6449bdf0c2d6) service to localhost/127.0.0.1:35787
2023-10-29 10:56:21,981 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/cluster_8c197d25-4fbf-3f92-e73f-8830694ec9c2/dfs/data/data3/current/BP-1345938757-172.31.14.131-1698576973645] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-10-29 10:56:21,981 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/cluster_8c197d25-4fbf-3f92-e73f-8830694ec9c2/dfs/data/data4/current/BP-1345938757-172.31.14.131-1698576973645] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-10-29 10:56:21,982 WARN  [Listener at localhost/38913] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called
2023-10-29 10:56:21,986 INFO  [Listener at localhost/38913] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2023-10-29 10:56:22,074 WARN  [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-10-29 10:56:22,099 WARN  [BP-1345938757-172.31.14.131-1698576973645 heartbeating to localhost/127.0.0.1:35787] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted
2023-10-29 10:56:22,099 WARN  [BP-1345938757-172.31.14.131-1698576973645 heartbeating to localhost/127.0.0.1:35787] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1345938757-172.31.14.131-1698576973645 (Datanode Uuid 9293b22f-87e1-47be-be13-eb912fda24ec) service to localhost/127.0.0.1:35787
2023-10-29 10:56:22,110 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/cluster_8c197d25-4fbf-3f92-e73f-8830694ec9c2/dfs/data/data1/current/BP-1345938757-172.31.14.131-1698576973645] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-10-29 10:56:22,110 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/124e0b94-c1d6-584a-ff3a-72a736d8f8fa/cluster_8c197d25-4fbf-3f92-e73f-8830694ec9c2/dfs/data/data2/current/BP-1345938757-172.31.14.131-1698576973645] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-10-29 10:56:22,143 INFO  [Listener at localhost/38913] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2023-10-29 10:56:22,299 INFO  [Listener at localhost/38913] hbase.HBaseTestingUtility(1293): Minicluster is down