2023-06-07 05:55:14,409 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c
2023-06-07 05:55:14,429 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStreamFSHLog timeout: 13 mins
2023-06-07 05:55:14,441 INFO  [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/cluster_fc4bd1dc-4af4-8692-7b68-48e08a3b1b99, deleteOnExit=true
2023-06-07 05:55:14,442 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/test.cache.data in system properties and HBase conf
2023-06-07 05:55:14,442 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/hadoop.tmp.dir in system properties and HBase conf
2023-06-07 05:55:14,443 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/hadoop.log.dir in system properties and HBase conf
2023-06-07 05:55:14,443 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/mapreduce.cluster.local.dir in system properties and HBase conf
2023-06-07 05:55:14,444 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/mapreduce.cluster.temp.dir in system properties and HBase conf
2023-06-07 05:55:14,444 INFO  [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF
2023-06-07 05:55:14,551 WARN  [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-06-07 05:55:14,933 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering
2023-06-07 05:55:14,936 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/yarn.node-labels.fs-store.root-dir in system properties and HBase conf
2023-06-07 05:55:14,937 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf
2023-06-07 05:55:14,937 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/yarn.nodemanager.log-dirs in system properties and HBase conf
2023-06-07 05:55:14,938 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf
2023-06-07 05:55:14,938 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf
2023-06-07 05:55:14,938 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf
2023-06-07 05:55:14,938 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf
2023-06-07 05:55:14,939 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/dfs.journalnode.edits.dir in system properties and HBase conf
2023-06-07 05:55:14,939 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf
2023-06-07 05:55:14,939 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/nfs.dump.dir in system properties and HBase conf
2023-06-07 05:55:14,939 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/java.io.tmpdir in system properties and HBase conf
2023-06-07 05:55:14,940 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/dfs.journalnode.edits.dir in system properties and HBase conf
2023-06-07 05:55:14,940 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf
2023-06-07 05:55:14,940 INFO  [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/fs.s3a.committer.staging.tmp.path in system properties and HBase conf
Formatting using clusterid: testClusterID
2023-06-07 05:55:15,446 WARN  [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS
2023-06-07 05:55:15,451 WARN  [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS
2023-06-07 05:55:15,729 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
2023-06-07 05:55:15,886 INFO  [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2023-06-07 05:55:15,901 WARN  [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j
2023-06-07 05:55:15,936 INFO  [Time-limited test] log.Slf4jLog(67): jetty-6.1.26
2023-06-07 05:55:15,968 INFO  [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/java.io.tmpdir/Jetty_localhost_41215_hdfs____.ne8n4t/webapp
2023-06-07 05:55:16,123 INFO  [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41215
2023-06-07 05:55:16,134 WARN  [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS
2023-06-07 05:55:16,134 WARN  [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS
2023-06-07 05:55:16,561 WARN  [Listener at localhost/41867] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j
2023-06-07 05:55:16,631 WARN  [Listener at localhost/41867] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS
2023-06-07 05:55:16,657 WARN  [Listener at localhost/41867] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j
2023-06-07 05:55:16,666 INFO  [Listener at localhost/41867] log.Slf4jLog(67): jetty-6.1.26
2023-06-07 05:55:16,673 INFO  [Listener at localhost/41867] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/java.io.tmpdir/Jetty_localhost_35985_datanode____ar0jzv/webapp
2023-06-07 05:55:16,772 INFO  [Listener at localhost/41867] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35985
2023-06-07 05:55:17,079 WARN  [Listener at localhost/34773] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j
2023-06-07 05:55:17,093 WARN  [Listener at localhost/34773] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS
2023-06-07 05:55:17,102 WARN  [Listener at localhost/34773] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j
2023-06-07 05:55:17,103 INFO  [Listener at localhost/34773] log.Slf4jLog(67): jetty-6.1.26
2023-06-07 05:55:17,108 INFO  [Listener at localhost/34773] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/java.io.tmpdir/Jetty_localhost_46751_datanode____jbe8c8/webapp
2023-06-07 05:55:17,205 INFO  [Listener at localhost/34773] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46751
2023-06-07 05:55:17,214 WARN  [Listener at localhost/39355] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j
2023-06-07 05:55:17,231 WARN  [Listener at localhost/39355] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS
2023-06-07 05:55:17,235 WARN  [Listener at localhost/39355] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j
2023-06-07 05:55:17,237 INFO  [Listener at localhost/39355] log.Slf4jLog(67): jetty-6.1.26
2023-06-07 05:55:17,245 INFO  [Listener at localhost/39355] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/java.io.tmpdir/Jetty_localhost_41119_datanode____.mej6al/webapp
2023-06-07 05:55:17,368 INFO  [Listener at localhost/39355] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41119
2023-06-07 05:55:17,376 WARN  [Listener at localhost/42253] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j
2023-06-07 05:55:17,580 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9502c60833c5d5c5: Processing first storage report for DS-d2c9157e-8773-47f8-91eb-0febe52a459e from datanode 21df1b83-f588-4c03-ab03-c28afab1ec23
2023-06-07 05:55:17,582 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9502c60833c5d5c5: from storage DS-d2c9157e-8773-47f8-91eb-0febe52a459e node DatanodeRegistration(127.0.0.1:42477, datanodeUuid=21df1b83-f588-4c03-ab03-c28afab1ec23, infoPort=37059, infoSecurePort=0, ipcPort=39355, storageInfo=lv=-57;cid=testClusterID;nsid=456584838;c=1686117315522), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0
2023-06-07 05:55:17,582 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8795d06dcdef0ffc: Processing first storage report for DS-2afa30db-26ce-4823-8294-10d440e640ec from datanode 1ff22bd5-4b17-4643-b09d-5db5b94b9eb7
2023-06-07 05:55:17,582 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8795d06dcdef0ffc: from storage DS-2afa30db-26ce-4823-8294-10d440e640ec node DatanodeRegistration(127.0.0.1:39199, datanodeUuid=1ff22bd5-4b17-4643-b09d-5db5b94b9eb7, infoPort=35073, infoSecurePort=0, ipcPort=34773, storageInfo=lv=-57;cid=testClusterID;nsid=456584838;c=1686117315522), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
2023-06-07 05:55:17,582 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x39d4533e6754bd7d: Processing first storage report for DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d from datanode fc06aca6-548a-457a-96b2-50139a675329
2023-06-07 05:55:17,582 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x39d4533e6754bd7d: from storage DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d node DatanodeRegistration(127.0.0.1:37745, datanodeUuid=fc06aca6-548a-457a-96b2-50139a675329, infoPort=46011, infoSecurePort=0, ipcPort=42253, storageInfo=lv=-57;cid=testClusterID;nsid=456584838;c=1686117315522), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
2023-06-07 05:55:17,583 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x9502c60833c5d5c5: Processing first storage report for DS-b1896bd7-407f-4683-887c-5a40a74cadfb from datanode 21df1b83-f588-4c03-ab03-c28afab1ec23
2023-06-07 05:55:17,583 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x9502c60833c5d5c5: from storage DS-b1896bd7-407f-4683-887c-5a40a74cadfb node DatanodeRegistration(127.0.0.1:42477, datanodeUuid=21df1b83-f588-4c03-ab03-c28afab1ec23, infoPort=37059, infoSecurePort=0, ipcPort=39355, storageInfo=lv=-57;cid=testClusterID;nsid=456584838;c=1686117315522), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
2023-06-07 05:55:17,583 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8795d06dcdef0ffc: Processing first storage report for DS-b15329bf-b3cc-4fad-8528-84bf579be1c6 from datanode 1ff22bd5-4b17-4643-b09d-5db5b94b9eb7
2023-06-07 05:55:17,583 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8795d06dcdef0ffc: from storage DS-b15329bf-b3cc-4fad-8528-84bf579be1c6 node DatanodeRegistration(127.0.0.1:39199, datanodeUuid=1ff22bd5-4b17-4643-b09d-5db5b94b9eb7, infoPort=35073, infoSecurePort=0, ipcPort=34773, storageInfo=lv=-57;cid=testClusterID;nsid=456584838;c=1686117315522), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
2023-06-07 05:55:17,583 INFO  [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x39d4533e6754bd7d: Processing first storage report for DS-a8bc8be2-a608-4c74-bdb4-0bd5dfea9ca2 from datanode fc06aca6-548a-457a-96b2-50139a675329
2023-06-07 05:55:17,583 INFO  [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x39d4533e6754bd7d: from storage DS-a8bc8be2-a608-4c74-bdb4-0bd5dfea9ca2 node DatanodeRegistration(127.0.0.1:37745, datanodeUuid=fc06aca6-548a-457a-96b2-50139a675329, infoPort=46011, infoSecurePort=0, ipcPort=42253, storageInfo=lv=-57;cid=testClusterID;nsid=456584838;c=1686117315522), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
2023-06-07 05:55:17,794 DEBUG [Listener at localhost/42253] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c
2023-06-07 05:55:17,812 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testLogrollWhileStreaming Thread=139, OpenFileDescriptor=402, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3292
2023-06-07 05:55:17,815 DEBUG [Listener at localhost/42253] util.ClassSize(228): Using Unsafe to estimate memory layout
2023-06-07 05:55:18,382 INFO  [Listener at localhost/42253] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-06-07 05:55:18,408 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:18,476 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testLogrollWhileStreaming, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testLogrollWhileStreaming, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:18,510 DEBUG [Listener at localhost/42253] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream
java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate()
	at java.lang.Class.getMethod(Class.java:1786)
	at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.<clinit>(CommonFSUtils.java:750)
	at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160)
	at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:307)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:70)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:881)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:574)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:515)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:160)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
	at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:295)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase.initWAL(WALEntryStreamTestBase.java:151)
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream.setUp(TestBasicWALEntryStream.java:80)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)
2023-06-07 05:55:18,730 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1141): Slow sync cost: 116 ms, current pipeline: []
2023-06-07 05:55:18,731 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1686117318508
2023-06-07 05:55:18,731 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK]]
2023-06-07 05:55:18,848 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:19,024 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1686117318508 with entries=3, filesize=437 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1686117319003
2023-06-07 05:55:19,026 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:19,026 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(716): hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1686117318508 is not closed yet, will try archiving it next time
2023-06-07 05:55:19,029 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:19,030 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:19,031 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 319
2023-06-07 05:55:19,131 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 319
2023-06-07 05:55:19,232 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 319
2023-06-07 05:55:19,333 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 319
2023-06-07 05:55:19,433 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 319
2023-06-07 05:55:19,534 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 319
2023-06-07 05:55:19,543 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:19,549 DEBUG [Listener at localhost/42253] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testLogrollWhileStreaming/testLogrollWhileStreaming.1686117318508
2023-06-07 05:55:19,559 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 201
2023-06-07 05:55:19,561 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testLogrollWhileStreaming
2023-06-07 05:55:19,598 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:19,599 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testLogrollWhileStreaming:(num 1686117319003)
2023-06-07 05:55:19,611 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testLogrollWhileStreaming Thread=146 (was 139)
Potentially hanging thread: IPC Parameter Sending Thread #1
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
	java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
	java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
	java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: LeaseRenewer:jenkins@localhost:41867
	java.lang.Thread.sleep(Native Method)
	org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411)
	org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76)
	org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: HBase-Metrics2-1
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:47902 [Waiting for operation #3]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@fef8f2e
	java.lang.Thread.sleep(Native Method)
	org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253)
	org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46)
	org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:36234 [Waiting for operation #3]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:58714 [Waiting for operation #4]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=447 (was 402) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3229 (was 3292)
2023-06-07 05:55:19,620 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testNewEntriesWhileStreaming Thread=146, OpenFileDescriptor=447, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3229
2023-06-07 05:55:19,622 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:19,629 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testNewEntriesWhileStreaming, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testNewEntriesWhileStreaming, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:19,646 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testNewEntriesWhileStreaming/testNewEntriesWhileStreaming.1686117319631
2023-06-07 05:55:19,646 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:19,661 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 201
2023-06-07 05:55:19,670 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 437
2023-06-07 05:55:19,671 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testNewEntriesWhileStreaming
2023-06-07 05:55:19,683 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:19,683 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testNewEntriesWhileStreaming:(num 1686117319631)
2023-06-07 05:55:19,693 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testNewEntriesWhileStreaming Thread=146 (was 146), OpenFileDescriptor=447 (was 447), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3225 (was 3229)
2023-06-07 05:55:19,703 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWithPartialWALEntryFailingFilter Thread=146, OpenFileDescriptor=447, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3224
2023-06-07 05:55:19,704 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:19,709 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderWithPartialWALEntryFailingFilter, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:19,751 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter.1686117319710
2023-06-07 05:55:19,751 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:19,920 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-06-07 05:55:19,936 WARN  [Thread-184] regionserver.ReplicationSourceWALReader(171): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$PartialWALEntryFailingWALEntryFilter.filter(TestBasicWALEntryStream.java:840)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:356)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:218)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:151)
2023-06-07 05:55:19,968 WARN  [Thread-184] regionserver.ReplicationSourceWALReader(171): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$PartialWALEntryFailingWALEntryFilter.filter(TestBasicWALEntryStream.java:840)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:356)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:218)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:151)
2023-06-07 05:55:20,011 WARN  [Thread-184] regionserver.ReplicationSourceWALReader(171): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$PartialWALEntryFailingWALEntryFilter.filter(TestBasicWALEntryStream.java:840)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:356)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:218)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:151)
2023-06-07 05:55:20,062 DEBUG [Thread-184] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:20,062 DEBUG [Thread-184] regionserver.ReplicationSourceWALReader(156): Read 3 WAL entries eligible for replication
2023-06-07 05:55:20,062 DEBUG [Thread-184] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:20,078 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter
2023-06-07 05:55:20,084 DEBUG [Thread-184] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:20,091 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:20,091 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderWithPartialWALEntryFailingFilter:(num 1686117319710)
2023-06-07 05:55:20,093 WARN  [Thread-184] regionserver.WALEntryStream(206): Couldn't get file length information about log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter.1686117319710, it was closed cleanly currently replicating from: hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter.1686117319710 at position: 458

2023-06-07 05:55:20,094 DEBUG [Thread-184] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter/testReplicationSourceWALReaderWithPartialWALEntryFailingFilter.1686117319710
2023-06-07 05:55:20,094 DEBUG [Thread-184] regionserver.ReplicationSourceWALReader(156): Read 0 WAL entries eligible for replication
2023-06-07 05:55:20,094 DEBUG [Thread-184] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:20,104 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWithPartialWALEntryFailingFilter Thread=147 (was 146)
Potentially hanging thread: Thread-184
	org.apache.hadoop.hdfs.client.impl.BlockReaderRemote2.read(BlockReaderRemote2.java:137)
	org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:822)
	org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:891)
	org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:950)
	org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:996)
	java.io.DataInputStream.readFully(DataInputStream.java:195)
	org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.setTrailerIfPresent(ProtobufLogReader.java:276)
	org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:238)
	org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.reset(ProtobufLogReader.java:160)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.resetReader(WALEntryStream.java:380)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:179)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:253)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:139)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:47902 [Waiting for operation #4]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:36234 [Waiting for operation #13]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:58714 [Waiting for operation #10]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=447 (was 447), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3201 (was 3224)
2023-06-07 05:55:20,113 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionForRecoveredQueueWithMultipleLogs Thread=147, OpenFileDescriptor=447, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3200
2023-06-07 05:55:20,115 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:20,120 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testEOFExceptionForRecoveredQueueWithMultipleLogs, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionForRecoveredQueueWithMultipleLogs, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:20,136 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionForRecoveredQueueWithMultipleLogs/testEOFExceptionForRecoveredQueueWithMultipleLogs.1686117320122
2023-06-07 05:55:20,136 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:20,362 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=10
2023-06-07 05:55:20,374 DEBUG [Thread-207] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/user/jenkins/log.1
2023-06-07 05:55:20,379 WARN  [Thread-207] regionserver.ReplicationSourceWALReader(297): Forcing removal of 0 length log in queue: hdfs://localhost:41867/user/jenkins/log.2
2023-06-07 05:55:20,379 DEBUG [Thread-207] regionserver.ReplicationSourceWALReader(325): Read 3 WAL entries eligible for replication
2023-06-07 05:55:20,379 DEBUG [Thread-207] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:20,380 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionForRecoveredQueueWithMultipleLogs
2023-06-07 05:55:20,390 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:20,390 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testEOFExceptionForRecoveredQueueWithMultipleLogs:(num 1686117320122)
2023-06-07 05:55:20,400 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionForRecoveredQueueWithMultipleLogs Thread=147 (was 147), OpenFileDescriptor=453 (was 447) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3178 (was 3200)
2023-06-07 05:55:20,409 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionInOldWALsDirectory Thread=147, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3178
2023-06-07 05:55:20,410 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:20,414 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testEOFExceptionInOldWALsDirectory, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:20,426 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320415
2023-06-07 05:55:20,427 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:20,441 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320415 with entries=0, filesize=83 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320427
2023-06-07 05:55:20,441 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK]]
2023-06-07 05:55:20,441 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(716): hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320415 is not closed yet, will try archiving it next time
2023-06-07 05:55:20,442 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:20,449 INFO  [WAL-Archive-0] wal.AbstractFSWAL(783): Archiving hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320415 to hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testEOFExceptionInOldWALsDirectory.1686117320415
2023-06-07 05:55:20,544 INFO  [Listener at localhost/42253] wal.AbstractFSWALProvider(464): Log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320415 was moved to hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testEOFExceptionInOldWALsDirectory.1686117320415
2023-06-07 05:55:20,557 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-06-07 05:55:20,558 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [10,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:20,561 INFO  [Thread-224] wal.AbstractFSWALProvider(464): Log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320415 was moved to hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testEOFExceptionInOldWALsDirectory.1686117320415
2023-06-07 05:55:20,565 INFO  [Thread-224] wal.AbstractFSWALProvider(464): Log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320415 was moved to hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testEOFExceptionInOldWALsDirectory.1686117320415
2023-06-07 05:55:20,565 WARN  [Thread-224] regionserver.ReplicationSourceWALReader(297): Forcing removal of 0 length log in queue: hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testEOFExceptionInOldWALsDirectory.1686117320415
2023-06-07 05:55:20,574 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-06-07 05:55:20,591 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-06-07 05:55:20,608 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-06-07 05:55:20,625 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-06-07 05:55:20,643 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-06-07 05:55:20,660 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory
2023-06-07 05:55:20,664 DEBUG [Thread-224] wal.ProtobufLogReader(420): EOF at position 83
2023-06-07 05:55:20,671 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:20,671 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testEOFExceptionInOldWALsDirectory:(num 1686117320427)
2023-06-07 05:55:20,672 WARN  [Thread-224] regionserver.WALEntryStream(206): Couldn't get file length information about log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320427, it was closed cleanly currently replicating from: hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320427 at position: 0

2023-06-07 05:55:20,673 DEBUG [Thread-224] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionInOldWALsDirectory/testEOFExceptionInOldWALsDirectory.1686117320427
2023-06-07 05:55:20,673 DEBUG [Thread-224] regionserver.ReplicationSourceWALReader(156): Read 0 WAL entries eligible for replication
2023-06-07 05:55:20,673 DEBUG [Thread-224] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:20,681 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionInOldWALsDirectory Thread=148 (was 147)
Potentially hanging thread: Thread-224
	com.sun.proxy.$Proxy31.getFileInfo(Unknown Source)
	org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:800)
	sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	com.sun.proxy.$Proxy34.getFileInfo(Unknown Source)
	org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
	org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1523)
	org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520)
	org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.checkAllBytesParsed(WALEntryStream.java:204)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:182)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:253)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:139)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:36234 [Waiting for operation #21]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:58714 [Waiting for operation #18]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3177 (was 3178)
2023-06-07 05:55:20,690 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderRecovered Thread=148, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3176
2023-06-07 05:55:20,691 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:20,695 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderRecovered, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderRecovered, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:20,704 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1686117320695
2023-06-07 05:55:20,705 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:20,721 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1686117320695 with entries=10, filesize=1.30 KB; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1686117320707
2023-06-07 05:55:20,721 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:20,722 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(716): hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1686117320695 is not closed yet, will try archiving it next time
2023-06-07 05:55:20,725 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderRecovered
2023-06-07 05:55:20,736 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=10, replicationBatchQueueCapacity=1
2023-06-07 05:55:20,745 DEBUG [Thread-247] regionserver.ReplicationSourceWALReader(156): Read 10 WAL entries eligible for replication
2023-06-07 05:55:20,749 DEBUG [Thread-247] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1686117320695
2023-06-07 05:55:20,754 DEBUG [Thread-247] regionserver.ReplicationSourceWALReader(156): Read 0 WAL entries eligible for replication
2023-06-07 05:55:20,759 DEBUG [Thread-247] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderRecovered/testReplicationSourceWALReaderRecovered.1686117320707
2023-06-07 05:55:20,759 DEBUG [Thread-247] regionserver.ReplicationSourceWALReader(156): Read 5 WAL entries eligible for replication
2023-06-07 05:55:20,759 DEBUG [Thread-247] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:20,763 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:20,764 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderRecovered:(num 1686117320707)
2023-06-07 05:55:20,773 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderRecovered Thread=148 (was 148), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3176 (was 3176)
2023-06-07 05:55:20,782 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionForRecoveredQueue Thread=148, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3176
2023-06-07 05:55:20,783 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:20,787 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testEOFExceptionForRecoveredQueue, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionForRecoveredQueue, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:20,799 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionForRecoveredQueue/testEOFExceptionForRecoveredQueue.1686117320788
2023-06-07 05:55:20,800 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:20,805 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=10
2023-06-07 05:55:20,809 WARN  [Thread-257] regionserver.ReplicationSourceWALReader(297): Forcing removal of 0 length log in queue: emptyLog
2023-06-07 05:55:20,809 DEBUG [Thread-257] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:20,810 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEOFExceptionForRecoveredQueue
2023-06-07 05:55:20,819 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:20,819 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testEOFExceptionForRecoveredQueue:(num 1686117320788)
2023-06-07 05:55:20,828 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEOFExceptionForRecoveredQueue Thread=148 (was 148), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3176 (was 3176)
2023-06-07 05:55:20,838 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWrongPosition Thread=148, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3175
2023-06-07 05:55:20,839 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:20,842 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderWrongPosition, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:20,855 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320844
2023-06-07 05:55:20,855 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:20,868 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320844 with entries=1, filesize=208 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320857
2023-06-07 05:55:20,870 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:20,871 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(716): hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320844 is not closed yet, will try archiving it next time
2023-06-07 05:55:20,879 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:20,882 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-06-07 05:55:20,892 DEBUG [Thread-274] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320844
2023-06-07 05:55:20,899 DEBUG [Thread-274] regionserver.ReplicationSourceWALReader(156): Read 1 WAL entries eligible for replication
2023-06-07 05:55:20,904 DEBUG [Thread-274] wal.ProtobufLogReader(420): EOF at position 2583
2023-06-07 05:55:20,904 DEBUG [Thread-274] regionserver.ReplicationSourceWALReader(156): Read 20 WAL entries eligible for replication
2023-06-07 05:55:20,904 DEBUG [Thread-274] wal.ProtobufLogReader(420): EOF at position 2583
2023-06-07 05:55:20,918 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320857 with entries=20, filesize=2.52 KB; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320904
2023-06-07 05:55:20,919 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK]]
2023-06-07 05:55:20,919 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(716): hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320857 is not closed yet, will try archiving it next time
2023-06-07 05:55:20,935 DEBUG [Thread-274] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320857
2023-06-07 05:55:20,943 DEBUG [Thread-274] regionserver.ReplicationSourceWALReader(156): Read 0 WAL entries eligible for replication
2023-06-07 05:55:20,946 DEBUG [Thread-274] wal.ProtobufLogReader(420): EOF at position 1333
2023-06-07 05:55:20,946 DEBUG [Thread-274] regionserver.ReplicationSourceWALReader(156): Read 10 WAL entries eligible for replication
2023-06-07 05:55:20,947 DEBUG [Thread-274] wal.ProtobufLogReader(420): EOF at position 1333
2023-06-07 05:55:20,947 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition
2023-06-07 05:55:20,962 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 3 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:20,962 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderWrongPosition:(num 1686117320904)
2023-06-07 05:55:20,971 INFO  [Thread-274] wal.AbstractFSWALProvider(464): Log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWrongPosition/testReplicationSourceWALReaderWrongPosition.1686117320904 was moved to hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testReplicationSourceWALReaderWrongPosition.1686117320904
2023-06-07 05:55:20,973 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWrongPosition Thread=149 (was 148)
Potentially hanging thread: Thread-274
	java.lang.Object.wait(Native Method)
	java.lang.Object.wait(Object.java:502)
	org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
	org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1528)
	org.apache.hadoop.ipc.Client.call(Client.java:1486)
	org.apache.hadoop.ipc.Client.call(Client.java:1385)
	org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
	com.sun.proxy.$Proxy31.getBlockLocations(Unknown Source)
	org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:263)
	sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	com.sun.proxy.$Proxy34.getBlockLocations(Unknown Source)
	org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:849)
	org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:838)
	org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:827)
	org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:332)
	org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:291)
	org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:276)
	org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1066)
	org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
	org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:323)
	org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
	org.apache.hadoop.fs.FileSystem.open(FileSystem.java:914)
	org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:214)
	org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.reset(ProtobufLogReader.java:160)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.resetReader(WALEntryStream.java:380)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:179)
	org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.tryAdvanceStreamAndCreateWALBatch(ReplicationSourceWALReader.java:253)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:139)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:47902 [Waiting for operation #11]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:36234 [Waiting for operation #24]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:58714 [Waiting for operation #24]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)
 - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3174 (was 3175)
2023-06-07 05:55:20,975 DEBUG [Thread-274] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testReplicationSourceWALReaderWrongPosition.1686117320904
2023-06-07 05:55:20,976 DEBUG [Thread-274] regionserver.ReplicationSourceWALReader(156): Read 0 WAL entries eligible for replication
2023-06-07 05:55:20,976 DEBUG [Thread-274] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:20,983 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testWALKeySerialization Thread=149, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3174
2023-06-07 05:55:20,984 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:20,988 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testWALKeySerialization, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testWALKeySerialization, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:21,003 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testWALKeySerialization/testWALKeySerialization.1686117320988
2023-06-07 05:55:21,003 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:21,007 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testWALKeySerialization
2023-06-07 05:55:21,015 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:21,015 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testWALKeySerialization:(num 1686117320988)
2023-06-07 05:55:21,025 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testWALKeySerialization Thread=149 (was 149), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3173 (was 3174)
2023-06-07 05:55:21,035 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderDisabled Thread=149, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3173
2023-06-07 05:55:21,036 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:21,040 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderDisabled, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderDisabled, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:21,053 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderDisabled/testReplicationSourceWALReaderDisabled.1686117321040
2023-06-07 05:55:21,053 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK]]
2023-06-07 05:55:21,069 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-06-07 05:55:21,071 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:21,185 DEBUG [Thread-302] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:21,186 DEBUG [Thread-302] regionserver.ReplicationSourceWALReader(156): Read 3 WAL entries eligible for replication
2023-06-07 05:55:21,186 DEBUG [Thread-302] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:21,186 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderDisabled
2023-06-07 05:55:21,194 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:21,194 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderDisabled:(num 1686117321040)
2023-06-07 05:55:21,199 INFO  [Thread-302] wal.AbstractFSWALProvider(464): Log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderDisabled/testReplicationSourceWALReaderDisabled.1686117321040 was moved to hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testReplicationSourceWALReaderDisabled.1686117321040
2023-06-07 05:55:21,207 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderDisabled Thread=151 (was 149)
Potentially hanging thread: Thread-302
	java.lang.Thread.sleep(Native Method)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.handleEmptyWALEntryBatch(ReplicationSourceWALReader.java:246)
	org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:142)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:36234 [Waiting for operation #27]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1109966468_17 at /127.0.0.1:58714 [Waiting for operation #26]
	sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
	org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	java.io.DataInputStream.readShort(DataInputStream.java:312)
	org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67)
	org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269)
	java.lang.Thread.run(Thread.java:750)

Potentially hanging thread: ForkJoinPool.commonPool-worker-1
	sun.misc.Unsafe.park(Native Method)
	java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
	java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
	java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
 - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3173 (was 3173)
2023-06-07 05:55:21,211 DEBUG [Thread-302] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testReplicationSourceWALReaderDisabled.1686117321040
2023-06-07 05:55:21,212 DEBUG [Thread-302] regionserver.ReplicationSourceWALReader(156): Read 0 WAL entries eligible for replication
2023-06-07 05:55:21,212 DEBUG [Thread-302] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:21,217 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testCleanClosedWALs Thread=151, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3173
2023-06-07 05:55:21,219 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:21,222 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testCleanClosedWALs, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testCleanClosedWALs, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:21,232 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testCleanClosedWALs/testCleanClosedWALs.1686117321222
2023-06-07 05:55:21,233 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:21,235 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:21,259 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testCleanClosedWALs/testCleanClosedWALs.1686117321222 with entries=1, filesize=208 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testCleanClosedWALs/testCleanClosedWALs.1686117321242
2023-06-07 05:55:21,259 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:21,259 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(716): hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testCleanClosedWALs/testCleanClosedWALs.1686117321222 is not closed yet, will try archiving it next time
2023-06-07 05:55:21,261 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:21,261 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 208
2023-06-07 05:55:21,362 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 208
2023-06-07 05:55:21,366 DEBUG [Listener at localhost/42253] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testCleanClosedWALs/testCleanClosedWALs.1686117321222
2023-06-07 05:55:21,374 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testCleanClosedWALs
2023-06-07 05:55:21,384 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:21,385 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testCleanClosedWALs:(num 1686117321242)
2023-06-07 05:55:21,395 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testCleanClosedWALs Thread=151 (was 151), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=254 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3172 (was 3173)
2023-06-07 05:55:21,404 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReadBeyondCommittedLength Thread=151, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=254, ProcessCount=175, AvailableMemoryMB=3172
2023-06-07 05:55:21,405 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:21,409 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReadBeyondCommittedLength, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReadBeyondCommittedLength, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:21,421 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReadBeyondCommittedLength/testReadBeyondCommittedLength.1686117321410
2023-06-07 05:55:21,421 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK]]
2023-06-07 05:55:21,432 DEBUG [Listener at localhost/42253] regionserver.WALEntryStream(265): The provider tells us the valid length for hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReadBeyondCommittedLength/testReadBeyondCommittedLength.1686117321410 is 318, but we have advanced to 319
2023-06-07 05:55:22,445 DEBUG [Listener at localhost/42253] regionserver.WALEntryStream(265): The provider tells us the valid length for hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReadBeyondCommittedLength/testReadBeyondCommittedLength.1686117321410 is 318, but we have advanced to 319
2023-06-07 05:55:22,458 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 319
2023-06-07 05:55:22,459 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReadBeyondCommittedLength
2023-06-07 05:55:22,466 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:22,467 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReadBeyondCommittedLength:(num 1686117321410)
2023-06-07 05:55:22,476 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReadBeyondCommittedLength Thread=150 (was 151), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 254), ProcessCount=175 (was 175), AvailableMemoryMB=3171 (was 3172)
2023-06-07 05:55:22,485 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testResumeStreamingFromPosition Thread=150, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=175, AvailableMemoryMB=3171
2023-06-07 05:55:22,486 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:22,490 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testResumeStreamingFromPosition, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testResumeStreamingFromPosition, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:22,501 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testResumeStreamingFromPosition/testResumeStreamingFromPosition.1686117322490
2023-06-07 05:55:22,502 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:22,520 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 437
2023-06-07 05:55:22,521 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testResumeStreamingFromPosition
2023-06-07 05:55:22,529 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:22,529 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testResumeStreamingFromPosition:(num 1686117322490)
2023-06-07 05:55:22,538 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testResumeStreamingFromPosition Thread=150 (was 150), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 242), ProcessCount=175 (was 175), AvailableMemoryMB=3171 (was 3171)
2023-06-07 05:55:22,547 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReader Thread=150, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=175, AvailableMemoryMB=3171
2023-06-07 05:55:22,548 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:22,551 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReader, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReader, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:22,562 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReader/testReplicationSourceWALReader.1686117322552
2023-06-07 05:55:22,562 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK]]
2023-06-07 05:55:22,575 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-06-07 05:55:22,585 DEBUG [Thread-354] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:22,585 DEBUG [Thread-354] regionserver.ReplicationSourceWALReader(156): Read 3 WAL entries eligible for replication
2023-06-07 05:55:22,586 DEBUG [Thread-354] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:22,603 DEBUG [Thread-354] wal.ProtobufLogReader(420): EOF at position 578
2023-06-07 05:55:22,603 DEBUG [Thread-354] regionserver.ReplicationSourceWALReader(156): Read 1 WAL entries eligible for replication
2023-06-07 05:55:22,603 DEBUG [Thread-354] wal.ProtobufLogReader(420): EOF at position 578
2023-06-07 05:55:22,604 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReader
2023-06-07 05:55:22,611 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:22,611 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReader:(num 1686117322552)
2023-06-07 05:55:22,615 INFO  [Thread-354] wal.AbstractFSWALProvider(464): Log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReader/testReplicationSourceWALReader.1686117322552 was moved to hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testReplicationSourceWALReader.1686117322552
2023-06-07 05:55:22,620 DEBUG [Thread-354] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testReplicationSourceWALReader.1686117322552
2023-06-07 05:55:22,621 DEBUG [Thread-354] regionserver.ReplicationSourceWALReader(156): Read 0 WAL entries eligible for replication
2023-06-07 05:55:22,621 DEBUG [Thread-354] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:22,624 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReader Thread=151 (was 150) - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 242), ProcessCount=175 (was 175), AvailableMemoryMB=3171 (was 3171)
2023-06-07 05:55:22,637 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWithFailingFilter Thread=151, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=175, AvailableMemoryMB=3171
2023-06-07 05:55:22,638 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:22,642 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testReplicationSourceWALReaderWithFailingFilter, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithFailingFilter, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:22,653 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithFailingFilter/testReplicationSourceWALReaderWithFailingFilter.1686117322643
2023-06-07 05:55:22,653 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:22,664 INFO  [Listener at localhost/42253] regionserver.ReplicationSourceWALReader(116): peerClusterZnode=null, ReplicationSourceWALReaderThread : null inited, replicationBatchSizeCapacity=67108864, replicationBatchCountCapacity=25000, replicationBatchQueueCapacity=1
2023-06-07 05:55:22,674 WARN  [Thread-366] regionserver.ReplicationSourceWALReader(171): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:356)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:218)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:151)
2023-06-07 05:55:22,701 WARN  [Thread-366] regionserver.ReplicationSourceWALReader(171): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:356)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:218)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:151)
2023-06-07 05:55:22,739 WARN  [Thread-366] regionserver.ReplicationSourceWALReader(171): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:356)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:218)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:151)
2023-06-07 05:55:22,787 WARN  [Thread-366] regionserver.ReplicationSourceWALReader(171): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:356)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:218)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:151)
2023-06-07 05:55:22,844 WARN  [Thread-366] regionserver.ReplicationSourceWALReader(171): Failed to read stream of replication entries
org.apache.hadoop.hbase.replication.regionserver.WALEntryFilterRetryableException: failing filter
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream$FailingWALEntryFilter.filter(TestBasicWALEntryStream.java:558)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.filterEntry(ReplicationSourceWALReader.java:356)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.readWALEntries(ReplicationSourceWALReader.java:218)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALReader.run(ReplicationSourceWALReader.java:151)
2023-06-07 05:55:22,914 DEBUG [Thread-366] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:22,914 DEBUG [Thread-366] regionserver.ReplicationSourceWALReader(156): Read 3 WAL entries eligible for replication
2023-06-07 05:55:22,914 DEBUG [Thread-366] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:22,915 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithFailingFilter
2023-06-07 05:55:22,921 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:22,921 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testReplicationSourceWALReaderWithFailingFilter:(num 1686117322643)
2023-06-07 05:55:22,926 INFO  [Thread-366] wal.AbstractFSWALProvider(464): Log hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testReplicationSourceWALReaderWithFailingFilter/testReplicationSourceWALReaderWithFailingFilter.1686117322643 was moved to hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testReplicationSourceWALReaderWithFailingFilter.1686117322643
2023-06-07 05:55:22,932 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testReplicationSourceWALReaderWithFailingFilter Thread=152 (was 151) - Thread LEAK? -, OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 242), ProcessCount=175 (was 175), AvailableMemoryMB=3170 (was 3171)
2023-06-07 05:55:22,934 DEBUG [Thread-366] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs/testReplicationSourceWALReaderWithFailingFilter.1686117322643
2023-06-07 05:55:22,935 DEBUG [Thread-366] regionserver.ReplicationSourceWALReader(156): Read 0 WAL entries eligible for replication
2023-06-07 05:55:22,935 DEBUG [Thread-366] regionserver.ReplicationSourceWALReader(241): Stopping the replication source wal reader
2023-06-07 05:55:22,942 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEmptyStream Thread=152, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=175, AvailableMemoryMB=3170
2023-06-07 05:55:22,943 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:22,946 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testEmptyStream, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEmptyStream, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:22,956 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEmptyStream/testEmptyStream.1686117322947
2023-06-07 05:55:22,956 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:22,963 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 83
2023-06-07 05:55:22,964 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testEmptyStream
2023-06-07 05:55:22,972 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:22,972 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testEmptyStream:(num 1686117322947)
2023-06-07 05:55:22,981 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testEmptyStream Thread=152 (was 152), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 242), ProcessCount=175 (was 175), AvailableMemoryMB=3170 (was 3170)
2023-06-07 05:55:22,990 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testAppendsWithRolls Thread=152, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=175, AvailableMemoryMB=3170
2023-06-07 05:55:22,991 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:22,994 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testAppendsWithRolls, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testAppendsWithRolls, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:23,003 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testAppendsWithRolls/testAppendsWithRolls.1686117322995
2023-06-07 05:55:23,003 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:23,010 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 208
2023-06-07 05:55:23,011 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 208
2023-06-07 05:55:23,011 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 208
2023-06-07 05:55:23,012 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:23,030 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testAppendsWithRolls/testAppendsWithRolls.1686117322995 with entries=3, filesize=458 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testAppendsWithRolls/testAppendsWithRolls.1686117323019
2023-06-07 05:55:23,030 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:23,030 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(716): hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testAppendsWithRolls/testAppendsWithRolls.1686117322995 is not closed yet, will try archiving it next time
2023-06-07 05:55:23,032 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:23,041 INFO  [Listener at localhost/42253] hbase.Waiter(180): Waiting up to [5,000] milli-secs(wait.for.ratio=[1])
2023-06-07 05:55:23,130 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(425): Encountered a malformed edit, seeking back to last good position in file, from 466 to 458
java.io.EOFException: Invalid PB, EOF? Ignoring; originalPosition=458, currentPosition=466
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:354)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:95)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:83)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:258)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.peek(WALEntryStream.java:110)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.next(WALEntryStream.java:117)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.access$001(WALEntryStreamTestBase.java:82)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.lambda$next$0(WALEntryStreamTestBase.java:95)
	at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:183)
	at org.apache.hadoop.hbase.Waiter.waitFor(Waiter.java:134)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStreamTestBase$WALEntryStreamWithRetries.next(WALEntryStreamTestBase.java:94)
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream.testAppendsWithRolls(TestBasicWALEntryStream.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: encoded_region_name, table_name, log_sequence_number, write_time
	at org.apache.hbase.thirdparty.com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:68)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:232)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:237)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.parseDelimitedFrom(ProtobufUtil.java:3576)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:348)
	... 40 more
2023-06-07 05:55:23,134 DEBUG [Listener at localhost/42253] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testAppendsWithRolls/testAppendsWithRolls.1686117322995
2023-06-07 05:55:23,139 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 208
2023-06-07 05:55:23,140 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testAppendsWithRolls
2023-06-07 05:55:23,147 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:23,147 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testAppendsWithRolls:(num 1686117323019)
2023-06-07 05:55:23,157 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testAppendsWithRolls Thread=152 (was 152), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 242), ProcessCount=175 (was 175), AvailableMemoryMB=3168 (was 3170)
2023-06-07 05:55:23,167 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testPosition Thread=152, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=175, AvailableMemoryMB=3168
2023-06-07 05:55:23,168 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:23,171 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testPosition, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testPosition, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:23,183 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testPosition/testPosition.1686117323172
2023-06-07 05:55:23,183 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK]]
2023-06-07 05:55:23,196 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 458
2023-06-07 05:55:23,196 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testPosition
2023-06-07 05:55:23,202 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:23,202 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testPosition:(num 1686117323172)
2023-06-07 05:55:23,211 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testPosition Thread=152 (was 152), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 242), ProcessCount=175 (was 175), AvailableMemoryMB=3168 (was 3168)
2023-06-07 05:55:23,220 INFO  [Listener at localhost/42253] hbase.ResourceChecker(147): before: replication.regionserver.TestBasicWALEntryStreamFSHLog#testSizeOfLogQueue Thread=152, OpenFileDescriptor=453, MaxFileDescriptor=60000, SystemLoadAverage=242, ProcessCount=175, AvailableMemoryMB=3168
2023-06-07 05:55:23,221 INFO  [Listener at localhost/42253] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider
2023-06-07 05:55:23,224 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=testSizeOfLogQueue, suffix=, logDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testSizeOfLogQueue, archiveDir=hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs, maxLogs=32
2023-06-07 05:55:23,234 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(806): New WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1686117323225
2023-06-07 05:55:23,234 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK]]
2023-06-07 05:55:23,246 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(802): Rolled WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1686117323225 with entries=1, filesize=208 B; new WAL /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1686117323236
2023-06-07 05:55:23,246 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(887): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37745,DS-5e5782fc-64ae-455f-94d3-6c0e694acb3d,DISK], DatanodeInfoWithStorage[127.0.0.1:39199,DS-2afa30db-26ce-4823-8294-10d440e640ec,DISK], DatanodeInfoWithStorage[127.0.0.1:42477,DS-d2c9157e-8773-47f8-91eb-0febe52a459e,DISK]]
2023-06-07 05:55:23,247 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(716): hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1686117323225 is not closed yet, will try archiving it next time
2023-06-07 05:55:23,254 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(425): Encountered a malformed edit, seeking back to last good position in file, from 216 to 208
java.io.EOFException: Invalid PB, EOF? Ignoring; originalPosition=208, currentPosition=216
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:354)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:95)
	at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:83)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.readNextEntryAndRecordReaderPosition(WALEntryStream.java:258)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.tryAdvanceEntry(WALEntryStream.java:172)
	at org.apache.hadoop.hbase.replication.regionserver.WALEntryStream.hasNext(WALEntryStream.java:101)
	at org.apache.hadoop.hbase.replication.regionserver.TestBasicWALEntryStream.testSizeOfLogQueue(TestBasicWALEntryStream.java:707)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: encoded_region_name, table_name, log_sequence_number, write_time
	at org.apache.hbase.thirdparty.com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:68)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:232)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:237)
	at org.apache.hbase.thirdparty.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
	at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.parseDelimitedFrom(ProtobufUtil.java:3576)
	at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:348)
	... 33 more
2023-06-07 05:55:23,257 DEBUG [Listener at localhost/42253] regionserver.WALEntryStream(247): EOF, closing hdfs://localhost:41867/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testSizeOfLogQueue/testSizeOfLogQueue.1686117323225
2023-06-07 05:55:23,262 DEBUG [Listener at localhost/42253] wal.ProtobufLogReader(420): EOF at position 83
2023-06-07 05:55:23,263 DEBUG [WAL-Shutdown-0] wal.FSHLog(489): Closing WAL writer in /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/WALs/testSizeOfLogQueue
2023-06-07 05:55:23,271 DEBUG [Listener at localhost/42253] wal.AbstractFSWAL(1028): Moved 2 WAL file(s) to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/oldWALs
2023-06-07 05:55:23,272 INFO  [Listener at localhost/42253] wal.AbstractFSWAL(1031): Closed WAL: FSHLog testSizeOfLogQueue:(num 1686117323236)
2023-06-07 05:55:23,283 INFO  [Listener at localhost/42253] hbase.ResourceChecker(175): after: replication.regionserver.TestBasicWALEntryStreamFSHLog#testSizeOfLogQueue Thread=152 (was 152), OpenFileDescriptor=453 (was 453), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=242 (was 242), ProcessCount=175 (was 175), AvailableMemoryMB=3168 (was 3168)
2023-06-07 05:55:23,283 INFO  [Listener at localhost/42253] hbase.HBaseTestingUtility(1286): Shutting down minicluster
2023-06-07 05:55:23,284 WARN  [Listener at localhost/42253] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called
2023-06-07 05:55:23,287 INFO  [Listener at localhost/42253] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2023-06-07 05:55:23,389 WARN  [BP-1659725854-172.31.14.131-1686117315522 heartbeating to localhost/127.0.0.1:41867] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted
2023-06-07 05:55:23,390 WARN  [BP-1659725854-172.31.14.131-1686117315522 heartbeating to localhost/127.0.0.1:41867] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1659725854-172.31.14.131-1686117315522 (Datanode Uuid fc06aca6-548a-457a-96b2-50139a675329) service to localhost/127.0.0.1:41867
2023-06-07 05:55:23,391 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/cluster_fc4bd1dc-4af4-8692-7b68-48e08a3b1b99/dfs/data/data5/current/BP-1659725854-172.31.14.131-1686117315522] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-06-07 05:55:23,392 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/cluster_fc4bd1dc-4af4-8692-7b68-48e08a3b1b99/dfs/data/data6/current/BP-1659725854-172.31.14.131-1686117315522] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-06-07 05:55:23,392 WARN  [Listener at localhost/42253] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called
2023-06-07 05:55:23,394 INFO  [Listener at localhost/42253] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2023-06-07 05:55:23,496 WARN  [BP-1659725854-172.31.14.131-1686117315522 heartbeating to localhost/127.0.0.1:41867] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted
2023-06-07 05:55:23,496 WARN  [BP-1659725854-172.31.14.131-1686117315522 heartbeating to localhost/127.0.0.1:41867] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1659725854-172.31.14.131-1686117315522 (Datanode Uuid 21df1b83-f588-4c03-ab03-c28afab1ec23) service to localhost/127.0.0.1:41867
2023-06-07 05:55:23,497 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/cluster_fc4bd1dc-4af4-8692-7b68-48e08a3b1b99/dfs/data/data3/current/BP-1659725854-172.31.14.131-1686117315522] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-06-07 05:55:23,498 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/cluster_fc4bd1dc-4af4-8692-7b68-48e08a3b1b99/dfs/data/data4/current/BP-1659725854-172.31.14.131-1686117315522] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-06-07 05:55:23,498 WARN  [Listener at localhost/42253] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called
2023-06-07 05:55:23,500 INFO  [Listener at localhost/42253] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2023-06-07 05:55:23,516 WARN  [BP-1659725854-172.31.14.131-1686117315522 heartbeating to localhost/127.0.0.1:41867] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1659725854-172.31.14.131-1686117315522 (Datanode Uuid 1ff22bd5-4b17-4643-b09d-5db5b94b9eb7) service to localhost/127.0.0.1:41867
2023-06-07 05:55:23,527 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/cluster_fc4bd1dc-4af4-8692-7b68-48e08a3b1b99/dfs/data/data1/current/BP-1659725854-172.31.14.131-1686117315522] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-06-07 05:55:23,527 WARN  [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/957dd107-516c-3160-1d0e-e3c0c514b37c/cluster_fc4bd1dc-4af4-8692-7b68-48e08a3b1b99/dfs/data/data2/current/BP-1659725854-172.31.14.131-1686117315522] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted
2023-06-07 05:55:23,630 INFO  [Listener at localhost/42253] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
2023-06-07 05:55:23,790 INFO  [Listener at localhost/42253] hbase.HBaseTestingUtility(1293): Minicluster is down