2023-07-20 21:00:18,607 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-20 21:00:18,795 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70 2023-07-20 21:00:18,806 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-20 21:00:18,807 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf, deleteOnExit=true 2023-07-20 21:00:18,807 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-20 21:00:18,807 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/test.cache.data in system properties and HBase conf 2023-07-20 21:00:18,807 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/hadoop.tmp.dir in system properties and HBase conf 2023-07-20 21:00:18,808 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/hadoop.log.dir in system properties and HBase conf 2023-07-20 21:00:18,808 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-20 21:00:18,809 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-20 21:00:18,809 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-20 21:00:18,902 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-20 21:00:19,218 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-20 21:00:19,221 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-20 21:00:19,221 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-20 21:00:19,222 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-20 21:00:19,222 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-20 21:00:19,222 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-20 21:00:19,222 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-20 21:00:19,223 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-20 21:00:19,223 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-20 21:00:19,224 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-20 21:00:19,224 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/nfs.dump.dir in system properties and HBase conf 2023-07-20 21:00:19,224 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/java.io.tmpdir in system properties and HBase conf 2023-07-20 21:00:19,224 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-20 21:00:19,225 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-20 21:00:19,225 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-20 21:00:19,650 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-20 21:00:19,653 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-20 21:00:20,058 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-20 21:00:20,192 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-20 21:00:20,205 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 21:00:20,236 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 21:00:20,263 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/java.io.tmpdir/Jetty_localhost_localdomain_39411_hdfs____.argdmg/webapp 2023-07-20 21:00:20,405 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:39411 2023-07-20 21:00:20,413 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-20 21:00:20,413 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-20 21:00:20,915 WARN [Listener at localhost.localdomain/36541] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 21:00:21,009 WARN [Listener at localhost.localdomain/36541] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-20 21:00:21,025 WARN [Listener at localhost.localdomain/36541] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 21:00:21,032 INFO [Listener at localhost.localdomain/36541] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 21:00:21,036 INFO [Listener at localhost.localdomain/36541] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/java.io.tmpdir/Jetty_localhost_42981_datanode____6u8rdn/webapp 2023-07-20 21:00:21,112 INFO [Listener at localhost.localdomain/36541] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42981 2023-07-20 21:00:21,405 WARN [Listener at localhost.localdomain/34557] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 21:00:21,467 WARN [Listener at localhost.localdomain/34557] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-20 21:00:21,472 WARN [Listener at localhost.localdomain/34557] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 21:00:21,475 INFO [Listener at localhost.localdomain/34557] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 21:00:21,483 INFO [Listener at localhost.localdomain/34557] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/java.io.tmpdir/Jetty_localhost_44359_datanode____.vba3nm/webapp 2023-07-20 21:00:21,573 INFO [Listener at localhost.localdomain/34557] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44359 2023-07-20 21:00:21,581 WARN [Listener at localhost.localdomain/46531] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 21:00:22,437 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5b7f1ce5b79759ad: Processing first storage report for DS-05acea34-49d1-4d24-8294-9af79e6a70d2 from datanode bd4412a0-8085-425f-b4c6-c07ddf89b733 2023-07-20 21:00:22,438 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5b7f1ce5b79759ad: from storage DS-05acea34-49d1-4d24-8294-9af79e6a70d2 node DatanodeRegistration(127.0.0.1:38951, datanodeUuid=bd4412a0-8085-425f-b4c6-c07ddf89b733, infoPort=42001, infoSecurePort=0, ipcPort=46531, storageInfo=lv=-57;cid=testClusterID;nsid=1426092689;c=1689886819716), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-20 21:00:22,438 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb249f3b77b36f061: Processing first storage report for DS-620eff30-290e-44ba-a990-7757dd03f2c2 from datanode d04461f5-d76d-414b-9379-b0c97d8b86c5 2023-07-20 21:00:22,438 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb249f3b77b36f061: from storage DS-620eff30-290e-44ba-a990-7757dd03f2c2 node DatanodeRegistration(127.0.0.1:45727, datanodeUuid=d04461f5-d76d-414b-9379-b0c97d8b86c5, infoPort=43827, infoSecurePort=0, ipcPort=34557, storageInfo=lv=-57;cid=testClusterID;nsid=1426092689;c=1689886819716), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-20 21:00:22,439 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5b7f1ce5b79759ad: Processing first storage report for DS-c82d5cfe-e0ff-464a-bf08-608028221a31 from datanode bd4412a0-8085-425f-b4c6-c07ddf89b733 2023-07-20 21:00:22,439 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5b7f1ce5b79759ad: from storage DS-c82d5cfe-e0ff-464a-bf08-608028221a31 node DatanodeRegistration(127.0.0.1:38951, datanodeUuid=bd4412a0-8085-425f-b4c6-c07ddf89b733, infoPort=42001, infoSecurePort=0, ipcPort=46531, storageInfo=lv=-57;cid=testClusterID;nsid=1426092689;c=1689886819716), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 21:00:22,439 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb249f3b77b36f061: Processing first storage report for DS-47fde804-1d8a-4b95-a780-094538711b9f from datanode d04461f5-d76d-414b-9379-b0c97d8b86c5 2023-07-20 21:00:22,439 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb249f3b77b36f061: from storage DS-47fde804-1d8a-4b95-a780-094538711b9f node DatanodeRegistration(127.0.0.1:45727, datanodeUuid=d04461f5-d76d-414b-9379-b0c97d8b86c5, infoPort=43827, infoSecurePort=0, ipcPort=34557, storageInfo=lv=-57;cid=testClusterID;nsid=1426092689;c=1689886819716), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 21:00:22,518 DEBUG [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70 2023-07-20 21:00:22,579 INFO [Listener at localhost.localdomain/46531] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/zookeeper_0, clientPort=57031, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-20 21:00:22,596 INFO [Listener at localhost.localdomain/46531] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57031 2023-07-20 21:00:22,605 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:22,608 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:23,242 INFO [Listener at localhost.localdomain/46531] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce with version=8 2023-07-20 21:00:23,243 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/hbase-staging 2023-07-20 21:00:23,511 INFO [Listener at localhost.localdomain/46531] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-20 21:00:23,866 INFO [Listener at localhost.localdomain/46531] client.ConnectionUtils(127): master/jenkins-hbase16:0 server-side Connection retries=45 2023-07-20 21:00:23,891 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:23,892 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:23,892 INFO [Listener at localhost.localdomain/46531] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 21:00:23,892 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:23,892 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 21:00:24,004 INFO [Listener at localhost.localdomain/46531] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 21:00:24,061 DEBUG [Listener at localhost.localdomain/46531] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-20 21:00:24,136 INFO [Listener at localhost.localdomain/46531] ipc.NettyRpcServer(120): Bind to /188.40.62.62:38535 2023-07-20 21:00:24,144 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:24,146 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:24,167 INFO [Listener at localhost.localdomain/46531] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38535 connecting to ZooKeeper ensemble=127.0.0.1:57031 2023-07-20 21:00:24,279 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:385350x0, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 21:00:24,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:38535-0x1018488ceeb0000 connected 2023-07-20 21:00:24,379 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 21:00:24,381 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 21:00:24,385 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 21:00:24,393 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38535 2023-07-20 21:00:24,394 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38535 2023-07-20 21:00:24,394 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38535 2023-07-20 21:00:24,395 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38535 2023-07-20 21:00:24,395 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38535 2023-07-20 21:00:24,400 INFO [Listener at localhost.localdomain/46531] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce, hbase.cluster.distributed=false 2023-07-20 21:00:24,456 INFO [Listener at localhost.localdomain/46531] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=45 2023-07-20 21:00:24,456 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:24,456 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:24,456 INFO [Listener at localhost.localdomain/46531] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 21:00:24,457 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:24,457 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 21:00:24,461 INFO [Listener at localhost.localdomain/46531] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 21:00:24,463 INFO [Listener at localhost.localdomain/46531] ipc.NettyRpcServer(120): Bind to /188.40.62.62:32921 2023-07-20 21:00:24,465 INFO [Listener at localhost.localdomain/46531] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 21:00:24,470 DEBUG [Listener at localhost.localdomain/46531] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 21:00:24,471 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:24,473 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:24,475 INFO [Listener at localhost.localdomain/46531] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:32921 connecting to ZooKeeper ensemble=127.0.0.1:57031 2023-07-20 21:00:24,486 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:329210x0, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 21:00:24,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:32921-0x1018488ceeb0001 connected 2023-07-20 21:00:24,487 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 21:00:24,489 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 21:00:24,490 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 21:00:24,491 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=32921 2023-07-20 21:00:24,491 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=32921 2023-07-20 21:00:24,492 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=32921 2023-07-20 21:00:24,493 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=32921 2023-07-20 21:00:24,493 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=32921 2023-07-20 21:00:24,510 INFO [Listener at localhost.localdomain/46531] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=45 2023-07-20 21:00:24,510 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:24,511 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:24,511 INFO [Listener at localhost.localdomain/46531] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 21:00:24,511 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:24,512 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 21:00:24,512 INFO [Listener at localhost.localdomain/46531] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 21:00:24,513 INFO [Listener at localhost.localdomain/46531] ipc.NettyRpcServer(120): Bind to /188.40.62.62:34051 2023-07-20 21:00:24,514 INFO [Listener at localhost.localdomain/46531] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 21:00:24,515 DEBUG [Listener at localhost.localdomain/46531] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 21:00:24,516 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:24,518 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:24,519 INFO [Listener at localhost.localdomain/46531] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34051 connecting to ZooKeeper ensemble=127.0.0.1:57031 2023-07-20 21:00:24,531 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:340510x0, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 21:00:24,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34051-0x1018488ceeb0002 connected 2023-07-20 21:00:24,533 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 21:00:24,534 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 21:00:24,535 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 21:00:24,535 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34051 2023-07-20 21:00:24,535 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34051 2023-07-20 21:00:24,536 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34051 2023-07-20 21:00:24,536 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34051 2023-07-20 21:00:24,537 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34051 2023-07-20 21:00:24,540 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:24,633 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-20 21:00:24,637 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:24,682 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 21:00:24,682 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 21:00:24,682 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 21:00:24,683 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:24,684 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 21:00:24,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 21:00:24,685 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase16.apache.org,38535,1689886823355 from backup master directory 2023-07-20 21:00:24,698 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:24,698 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-20 21:00:24,699 WARN [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 21:00:24,699 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:24,703 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-20 21:00:24,706 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-20 21:00:24,786 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/hbase.id with ID: dbbe0cde-59ef-4799-85ef-0469c06fee88 2023-07-20 21:00:24,828 INFO [master/jenkins-hbase16:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:24,852 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:24,897 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x58dfe532 to 127.0.0.1:57031 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 21:00:24,937 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c2b0df3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 21:00:24,955 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-20 21:00:24,957 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-20 21:00:24,971 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-20 21:00:24,971 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-20 21:00:24,973 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-20 21:00:24,976 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-20 21:00:24,977 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 21:00:25,004 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store-tmp 2023-07-20 21:00:25,035 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:25,035 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-20 21:00:25,035 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 21:00:25,035 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 21:00:25,035 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-20 21:00:25,035 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 21:00:25,036 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 21:00:25,036 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 21:00:25,037 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/WALs/jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:25,057 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C38535%2C1689886823355, suffix=, logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/WALs/jenkins-hbase16.apache.org,38535,1689886823355, archiveDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/oldWALs, maxLogs=10 2023-07-20 21:00:25,106 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK] 2023-07-20 21:00:25,106 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK] 2023-07-20 21:00:25,113 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 21:00:25,169 INFO [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/WALs/jenkins-hbase16.apache.org,38535,1689886823355/jenkins-hbase16.apache.org%2C38535%2C1689886823355.1689886825066 2023-07-20 21:00:25,170 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK], DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK]] 2023-07-20 21:00:25,170 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:25,171 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:25,173 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 21:00:25,174 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 21:00:25,222 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-20 21:00:25,228 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-20 21:00:25,247 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-20 21:00:25,257 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:25,262 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-20 21:00:25,264 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-20 21:00:25,278 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 21:00:25,283 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 21:00:25,284 INFO [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60461072, jitterRate=-0.0990598201751709}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 21:00:25,285 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 21:00:25,286 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-20 21:00:25,302 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-20 21:00:25,302 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-20 21:00:25,304 INFO [master/jenkins-hbase16:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-20 21:00:25,306 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-20 21:00:25,331 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-07-20 21:00:25,332 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-20 21:00:25,352 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-20 21:00:25,357 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-20 21:00:25,377 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-20 21:00:25,380 INFO [master/jenkins-hbase16:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-20 21:00:25,381 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-20 21:00:25,385 INFO [master/jenkins-hbase16:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-20 21:00:25,389 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-20 21:00:25,427 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:25,429 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-20 21:00:25,429 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-20 21:00:25,441 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-20 21:00:25,452 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 21:00:25,452 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 21:00:25,452 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 21:00:25,452 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:25,453 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase16.apache.org,38535,1689886823355, sessionid=0x1018488ceeb0000, setting cluster-up flag (Was=false) 2023-07-20 21:00:25,481 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:25,506 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-20 21:00:25,508 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:25,527 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:25,552 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-20 21:00:25,554 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:25,557 WARN [master/jenkins-hbase16:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.hbase-snapshot/.tmp 2023-07-20 21:00:25,640 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(951): ClusterId : dbbe0cde-59ef-4799-85ef-0469c06fee88 2023-07-20 21:00:25,640 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(951): ClusterId : dbbe0cde-59ef-4799-85ef-0469c06fee88 2023-07-20 21:00:25,645 DEBUG [RS:0;jenkins-hbase16:32921] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 21:00:25,645 DEBUG [RS:1;jenkins-hbase16:34051] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 21:00:25,655 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-20 21:00:25,658 DEBUG [RS:1;jenkins-hbase16:34051] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 21:00:25,659 DEBUG [RS:1;jenkins-hbase16:34051] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 21:00:25,658 DEBUG [RS:0;jenkins-hbase16:32921] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 21:00:25,659 DEBUG [RS:0;jenkins-hbase16:32921] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 21:00:25,665 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-07-20 21:00:25,665 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-07-20 21:00:25,665 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-07-20 21:00:25,665 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=5, maxPoolSize=5 2023-07-20 21:00:25,665 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase16:0, corePoolSize=10, maxPoolSize=10 2023-07-20 21:00:25,665 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,665 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-07-20 21:00:25,666 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,667 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689886855667 2023-07-20 21:00:25,669 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-20 21:00:25,671 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-20 21:00:25,671 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-20 21:00:25,675 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-20 21:00:25,678 DEBUG [RS:0;jenkins-hbase16:32921] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 21:00:25,678 DEBUG [RS:1;jenkins-hbase16:34051] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 21:00:25,680 DEBUG [RS:0;jenkins-hbase16:32921] zookeeper.ReadOnlyZKClient(139): Connect 0x5cd9e5fe to 127.0.0.1:57031 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 21:00:25,680 DEBUG [RS:1;jenkins-hbase16:34051] zookeeper.ReadOnlyZKClient(139): Connect 0x28fd7dde to 127.0.0.1:57031 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 21:00:25,683 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-20 21:00:25,692 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-20 21:00:25,693 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-20 21:00:25,693 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-20 21:00:25,693 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-20 21:00:25,694 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,695 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-20 21:00:25,697 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-20 21:00:25,697 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-20 21:00:25,699 DEBUG [RS:1;jenkins-hbase16:34051] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@263a86b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 21:00:25,699 DEBUG [RS:0;jenkins-hbase16:32921] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@521e0015, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 21:00:25,700 DEBUG [RS:1;jenkins-hbase16:34051] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@220516ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-07-20 21:00:25,700 DEBUG [RS:0;jenkins-hbase16:32921] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56218e8e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-07-20 21:00:25,700 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-20 21:00:25,701 INFO [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-20 21:00:25,707 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1689886825705,5,FailOnTimeoutGroup] 2023-07-20 21:00:25,707 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1689886825707,5,FailOnTimeoutGroup] 2023-07-20 21:00:25,707 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,708 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-20 21:00:25,709 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,709 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,724 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-20 21:00:25,725 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-20 21:00:25,726 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce 2023-07-20 21:00:25,729 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase16:32921 2023-07-20 21:00:25,729 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase16:34051 2023-07-20 21:00:25,734 INFO [RS:1;jenkins-hbase16:34051] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 21:00:25,734 INFO [RS:0;jenkins-hbase16:32921] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 21:00:25,735 INFO [RS:0;jenkins-hbase16:32921] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 21:00:25,735 INFO [RS:1;jenkins-hbase16:34051] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 21:00:25,735 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 21:00:25,735 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 21:00:25,738 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase16.apache.org,38535,1689886823355 with isa=jenkins-hbase16.apache.org/188.40.62.62:34051, startcode=1689886824509 2023-07-20 21:00:25,738 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase16.apache.org,38535,1689886823355 with isa=jenkins-hbase16.apache.org/188.40.62.62:32921, startcode=1689886824455 2023-07-20 21:00:25,749 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:25,753 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 21:00:25,756 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info 2023-07-20 21:00:25,757 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 21:00:25,758 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:25,759 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 21:00:25,762 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/rep_barrier 2023-07-20 21:00:25,762 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 21:00:25,763 DEBUG [RS:0;jenkins-hbase16:32921] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 21:00:25,763 DEBUG [RS:1;jenkins-hbase16:34051] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 21:00:25,764 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:25,764 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 21:00:25,766 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table 2023-07-20 21:00:25,767 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 21:00:25,768 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:25,769 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740 2023-07-20 21:00:25,770 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740 2023-07-20 21:00:25,774 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 21:00:25,776 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 21:00:25,779 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 21:00:25,780 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59735507, jitterRate=-0.10987158119678497}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 21:00:25,781 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 21:00:25,781 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 21:00:25,781 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 21:00:25,781 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 21:00:25,781 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 21:00:25,781 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 21:00:25,783 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 21:00:25,783 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 21:00:25,799 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-20 21:00:25,799 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-20 21:00:25,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 21:00:25,816 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:47475, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 21:00:25,816 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57307, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 21:00:25,821 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 21:00:25,824 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-20 21:00:25,828 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38535] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:25,838 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38535] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:25,842 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce 2023-07-20 21:00:25,843 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:36541 2023-07-20 21:00:25,843 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 21:00:25,842 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce 2023-07-20 21:00:25,843 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:36541 2023-07-20 21:00:25,843 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 21:00:25,883 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:25,884 DEBUG [RS:1;jenkins-hbase16:34051] zookeeper.ZKUtil(162): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:25,884 DEBUG [RS:0;jenkins-hbase16:32921] zookeeper.ZKUtil(162): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:25,884 WARN [RS:1;jenkins-hbase16:34051] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 21:00:25,885 WARN [RS:0;jenkins-hbase16:32921] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 21:00:25,885 INFO [RS:1;jenkins-hbase16:34051] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 21:00:25,886 INFO [RS:0;jenkins-hbase16:32921] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 21:00:25,886 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:25,887 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:25,887 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,32921,1689886824455] 2023-07-20 21:00:25,888 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,34051,1689886824509] 2023-07-20 21:00:25,903 DEBUG [RS:1;jenkins-hbase16:34051] zookeeper.ZKUtil(162): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:25,903 DEBUG [RS:0;jenkins-hbase16:32921] zookeeper.ZKUtil(162): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:25,903 DEBUG [RS:1;jenkins-hbase16:34051] zookeeper.ZKUtil(162): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:25,903 DEBUG [RS:0;jenkins-hbase16:32921] zookeeper.ZKUtil(162): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:25,914 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 21:00:25,914 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 21:00:25,923 INFO [RS:1;jenkins-hbase16:34051] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 21:00:25,923 INFO [RS:0;jenkins-hbase16:32921] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 21:00:25,943 INFO [RS:1;jenkins-hbase16:34051] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 21:00:25,943 INFO [RS:0;jenkins-hbase16:32921] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 21:00:25,947 INFO [RS:1;jenkins-hbase16:34051] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 21:00:25,947 INFO [RS:0;jenkins-hbase16:32921] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 21:00:25,947 INFO [RS:1;jenkins-hbase16:34051] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,947 INFO [RS:0;jenkins-hbase16:32921] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,949 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 21:00:25,949 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 21:00:25,955 INFO [RS:1;jenkins-hbase16:34051] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,955 INFO [RS:0;jenkins-hbase16:32921] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,955 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,955 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,955 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-07-20 21:00:25,956 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-07-20 21:00:25,956 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,956 DEBUG [RS:1;jenkins-hbase16:34051] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,957 DEBUG [RS:0;jenkins-hbase16:32921] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:25,958 INFO [RS:1;jenkins-hbase16:34051] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,958 INFO [RS:0;jenkins-hbase16:32921] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,958 INFO [RS:1;jenkins-hbase16:34051] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,958 INFO [RS:0;jenkins-hbase16:32921] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,958 INFO [RS:1;jenkins-hbase16:34051] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,958 INFO [RS:0;jenkins-hbase16:32921] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,972 INFO [RS:1;jenkins-hbase16:34051] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 21:00:25,972 INFO [RS:0;jenkins-hbase16:32921] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 21:00:25,974 INFO [RS:0;jenkins-hbase16:32921] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,32921,1689886824455-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,974 INFO [RS:1;jenkins-hbase16:34051] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,34051,1689886824509-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:25,976 DEBUG [jenkins-hbase16:38535] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-20 21:00:25,980 DEBUG [jenkins-hbase16:38535] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-07-20 21:00:25,986 DEBUG [jenkins-hbase16:38535] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 21:00:25,987 DEBUG [jenkins-hbase16:38535] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 21:00:25,987 DEBUG [jenkins-hbase16:38535] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 21:00:25,990 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,34051,1689886824509, state=OPENING 2023-07-20 21:00:25,990 INFO [RS:0;jenkins-hbase16:32921] regionserver.Replication(203): jenkins-hbase16.apache.org,32921,1689886824455 started 2023-07-20 21:00:25,991 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1637): Serving as jenkins-hbase16.apache.org,32921,1689886824455, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:32921, sessionid=0x1018488ceeb0001 2023-07-20 21:00:25,991 INFO [RS:1;jenkins-hbase16:34051] regionserver.Replication(203): jenkins-hbase16.apache.org,34051,1689886824509 started 2023-07-20 21:00:25,991 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1637): Serving as jenkins-hbase16.apache.org,34051,1689886824509, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:34051, sessionid=0x1018488ceeb0002 2023-07-20 21:00:25,991 DEBUG [RS:0;jenkins-hbase16:32921] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 21:00:25,991 DEBUG [RS:1;jenkins-hbase16:34051] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 21:00:25,991 DEBUG [RS:0;jenkins-hbase16:32921] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:25,991 DEBUG [RS:1;jenkins-hbase16:34051] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:25,992 DEBUG [RS:0;jenkins-hbase16:32921] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,32921,1689886824455' 2023-07-20 21:00:25,992 DEBUG [RS:1;jenkins-hbase16:34051] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,34051,1689886824509' 2023-07-20 21:00:25,992 DEBUG [RS:1;jenkins-hbase16:34051] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 21:00:25,992 DEBUG [RS:0;jenkins-hbase16:32921] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 21:00:25,993 DEBUG [RS:1;jenkins-hbase16:34051] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 21:00:25,993 DEBUG [RS:0;jenkins-hbase16:32921] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 21:00:25,994 DEBUG [RS:1;jenkins-hbase16:34051] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 21:00:25,994 DEBUG [RS:0;jenkins-hbase16:32921] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 21:00:25,994 DEBUG [RS:1;jenkins-hbase16:34051] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 21:00:25,994 DEBUG [RS:0;jenkins-hbase16:32921] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 21:00:25,994 DEBUG [RS:1;jenkins-hbase16:34051] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:25,994 DEBUG [RS:0;jenkins-hbase16:32921] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:25,994 DEBUG [RS:1;jenkins-hbase16:34051] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,34051,1689886824509' 2023-07-20 21:00:25,994 DEBUG [RS:0;jenkins-hbase16:32921] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,32921,1689886824455' 2023-07-20 21:00:25,994 DEBUG [RS:1;jenkins-hbase16:34051] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 21:00:25,995 DEBUG [RS:0;jenkins-hbase16:32921] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 21:00:25,995 DEBUG [RS:1;jenkins-hbase16:34051] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 21:00:25,995 DEBUG [RS:0;jenkins-hbase16:32921] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 21:00:25,995 DEBUG [RS:1;jenkins-hbase16:34051] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 21:00:25,995 DEBUG [RS:0;jenkins-hbase16:32921] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 21:00:25,995 INFO [RS:1;jenkins-hbase16:34051] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 21:00:25,995 INFO [RS:0;jenkins-hbase16:32921] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 21:00:25,996 INFO [RS:1;jenkins-hbase16:34051] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 21:00:25,996 INFO [RS:0;jenkins-hbase16:32921] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 21:00:26,011 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-20 21:00:26,019 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:26,020 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 21:00:26,025 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,34051,1689886824509}] 2023-07-20 21:00:26,112 INFO [RS:0;jenkins-hbase16:32921] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C32921%2C1689886824455, suffix=, logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455, archiveDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs, maxLogs=32 2023-07-20 21:00:26,112 INFO [RS:1;jenkins-hbase16:34051] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C34051%2C1689886824509, suffix=, logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509, archiveDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs, maxLogs=32 2023-07-20 21:00:26,131 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK] 2023-07-20 21:00:26,131 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK] 2023-07-20 21:00:26,131 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK] 2023-07-20 21:00:26,132 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK] 2023-07-20 21:00:26,140 INFO [RS:1;jenkins-hbase16:34051] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115 2023-07-20 21:00:26,140 INFO [RS:0;jenkins-hbase16:32921] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115 2023-07-20 21:00:26,141 DEBUG [RS:1;jenkins-hbase16:34051] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK], DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK]] 2023-07-20 21:00:26,141 DEBUG [RS:0;jenkins-hbase16:32921] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK], DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK]] 2023-07-20 21:00:26,214 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:26,216 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 21:00:26,219 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:49740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 21:00:26,232 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 21:00:26,233 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 21:00:26,237 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509, archiveDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs, maxLogs=32 2023-07-20 21:00:26,254 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK] 2023-07-20 21:00:26,254 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK] 2023-07-20 21:00:26,262 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta 2023-07-20 21:00:26,265 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK], DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK]] 2023-07-20 21:00:26,266 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:26,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 21:00:26,283 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 21:00:26,309 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 21:00:26,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 21:00:26,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:26,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 21:00:26,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 21:00:26,318 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 21:00:26,320 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info 2023-07-20 21:00:26,320 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info 2023-07-20 21:00:26,320 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 21:00:26,321 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:26,322 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 21:00:26,323 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/rep_barrier 2023-07-20 21:00:26,323 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/rep_barrier 2023-07-20 21:00:26,324 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 21:00:26,325 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:26,325 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 21:00:26,327 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table 2023-07-20 21:00:26,327 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table 2023-07-20 21:00:26,328 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 21:00:26,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:26,332 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740 2023-07-20 21:00:26,335 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740 2023-07-20 21:00:26,338 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 21:00:26,340 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 21:00:26,341 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69563092, jitterRate=0.036570847034454346}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 21:00:26,342 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 21:00:26,349 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689886826208 2023-07-20 21:00:26,364 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 21:00:26,365 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 21:00:26,366 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,34051,1689886824509, state=OPEN 2023-07-20 21:00:26,402 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 21:00:26,403 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 21:00:26,409 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-20 21:00:26,409 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,34051,1689886824509 in 378 msec 2023-07-20 21:00:26,418 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-20 21:00:26,418 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 603 msec 2023-07-20 21:00:26,423 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 822 msec 2023-07-20 21:00:26,424 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689886826424, completionTime=-1 2023-07-20 21:00:26,424 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-20 21:00:26,424 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-20 21:00:26,481 DEBUG [hconnection-0x11f37f3f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:26,484 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:49742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:26,499 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-20 21:00:26,499 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689886886499 2023-07-20 21:00:26,499 INFO [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689886946499 2023-07-20 21:00:26,499 INFO [master/jenkins-hbase16:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 75 msec 2023-07-20 21:00:26,534 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,38535,1689886823355-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:26,535 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,38535,1689886823355-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:26,535 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,38535,1689886823355-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:26,537 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase16:38535, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:26,537 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:26,544 DEBUG [master/jenkins-hbase16:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-20 21:00:26,551 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-20 21:00:26,553 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-20 21:00:26,560 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-20 21:00:26,563 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-20 21:00:26,565 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-20 21:00:26,586 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:26,589 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b empty. 2023-07-20 21:00:26,590 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:26,591 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-20 21:00:26,621 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-20 21:00:26,624 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => e6b0db802b49804f36d630fc7a235f8b, NAME => 'hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp 2023-07-20 21:00:26,638 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:26,638 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing e6b0db802b49804f36d630fc7a235f8b, disabling compactions & flushes 2023-07-20 21:00:26,638 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:26,638 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:26,638 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. after waiting 0 ms 2023-07-20 21:00:26,638 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:26,638 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:26,638 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for e6b0db802b49804f36d630fc7a235f8b: 2023-07-20 21:00:26,643 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-20 21:00:26,658 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689886826646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689886826646"}]},"ts":"1689886826646"} 2023-07-20 21:00:26,683 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-20 21:00:26,685 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-20 21:00:26,690 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689886826685"}]},"ts":"1689886826685"} 2023-07-20 21:00:26,694 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-20 21:00:26,715 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-07-20 21:00:26,718 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 21:00:26,718 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 21:00:26,719 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 21:00:26,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN}] 2023-07-20 21:00:26,727 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN 2023-07-20 21:00:26,729 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34051,1689886824509; forceNewPlan=false, retain=false 2023-07-20 21:00:26,885 INFO [jenkins-hbase16:38535] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-20 21:00:26,890 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e6b0db802b49804f36d630fc7a235f8b, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:26,890 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689886826888"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689886826888"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689886826888"}]},"ts":"1689886826888"} 2023-07-20 21:00:26,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e6b0db802b49804f36d630fc7a235f8b, server=jenkins-hbase16.apache.org,34051,1689886824509}] 2023-07-20 21:00:27,062 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:27,063 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e6b0db802b49804f36d630fc7a235f8b, NAME => 'hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:27,064 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:27,064 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:27,064 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:27,064 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:27,066 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:27,068 DEBUG [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info 2023-07-20 21:00:27,069 DEBUG [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info 2023-07-20 21:00:27,069 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6b0db802b49804f36d630fc7a235f8b columnFamilyName info 2023-07-20 21:00:27,070 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] regionserver.HStore(310): Store=e6b0db802b49804f36d630fc7a235f8b/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:27,071 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:27,072 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:27,076 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:27,079 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 21:00:27,080 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened e6b0db802b49804f36d630fc7a235f8b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59966302, jitterRate=-0.1064324676990509}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 21:00:27,080 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for e6b0db802b49804f36d630fc7a235f8b: 2023-07-20 21:00:27,082 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b., pid=6, masterSystemTime=1689886827052 2023-07-20 21:00:27,086 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:27,087 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:27,089 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e6b0db802b49804f36d630fc7a235f8b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:27,089 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689886827088"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689886827088"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689886827088"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689886827088"}]},"ts":"1689886827088"} 2023-07-20 21:00:27,098 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-20 21:00:27,099 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e6b0db802b49804f36d630fc7a235f8b, server=jenkins-hbase16.apache.org,34051,1689886824509 in 197 msec 2023-07-20 21:00:27,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-20 21:00:27,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN in 377 msec 2023-07-20 21:00:27,103 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-20 21:00:27,104 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689886827104"}]},"ts":"1689886827104"} 2023-07-20 21:00:27,107 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-20 21:00:27,116 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-20 21:00:27,120 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 564 msec 2023-07-20 21:00:27,163 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-20 21:00:27,173 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-20 21:00:27,173 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:27,210 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-20 21:00:27,235 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-20 21:00:27,249 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 45 msec 2023-07-20 21:00:27,253 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-20 21:00:27,269 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-20 21:00:27,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 27 msec 2023-07-20 21:00:27,302 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-20 21:00:27,319 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-20 21:00:27,319 INFO [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.620sec 2023-07-20 21:00:27,323 INFO [master/jenkins-hbase16:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-20 21:00:27,325 INFO [master/jenkins-hbase16:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-20 21:00:27,326 INFO [master/jenkins-hbase16:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-20 21:00:27,327 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,38535,1689886823355-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-20 21:00:27,328 INFO [master/jenkins-hbase16:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,38535,1689886823355-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-20 21:00:27,336 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-20 21:00:27,347 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(139): Connect 0x62eb3340 to 127.0.0.1:57031 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 21:00:27,357 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1778e62d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 21:00:27,373 DEBUG [hconnection-0x354830d7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:27,385 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:49756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:27,395 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:27,406 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-20 21:00:27,409 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:57764, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-20 21:00:27,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.HMaster$4(2112): Client=jenkins//188.40.62.62 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-20 21:00:27,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-20 21:00:27,426 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-20 21:00:27,428 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-20 21:00:27,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.MasterRpcServices(700): Client=jenkins//188.40.62.62 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-20 21:00:27,431 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp/data/default/t/2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,432 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp/data/default/t/2587dac91101456d664c734eee8a5428 empty. 2023-07-20 21:00:27,434 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp/data/default/t/2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,434 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-20 21:00:27,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 21:00:27,453 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-20 21:00:27,455 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2587dac91101456d664c734eee8a5428, NAME => 't,,1689886827417.2587dac91101456d664c734eee8a5428.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/.tmp 2023-07-20 21:00:27,470 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689886827417.2587dac91101456d664c734eee8a5428.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:27,470 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 2587dac91101456d664c734eee8a5428, disabling compactions & flushes 2023-07-20 21:00:27,470 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:27,470 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:27,470 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689886827417.2587dac91101456d664c734eee8a5428. after waiting 0 ms 2023-07-20 21:00:27,470 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:27,470 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:27,470 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 2587dac91101456d664c734eee8a5428: 2023-07-20 21:00:27,476 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-20 21:00:27,478 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689886827417.2587dac91101456d664c734eee8a5428.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689886827478"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689886827478"}]},"ts":"1689886827478"} 2023-07-20 21:00:27,481 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-20 21:00:27,482 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-20 21:00:27,482 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689886827482"}]},"ts":"1689886827482"} 2023-07-20 21:00:27,484 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-20 21:00:27,503 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase16.apache.org=0} racks are {/default-rack=0} 2023-07-20 21:00:27,505 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 21:00:27,505 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 21:00:27,505 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 21:00:27,505 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN}] 2023-07-20 21:00:27,509 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN 2023-07-20 21:00:27,511 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN; state=OFFLINE, location=jenkins-hbase16.apache.org,34051,1689886824509; forceNewPlan=false, retain=false 2023-07-20 21:00:27,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 21:00:27,661 INFO [jenkins-hbase16:38535] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-20 21:00:27,663 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2587dac91101456d664c734eee8a5428, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:27,663 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689886827417.2587dac91101456d664c734eee8a5428.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689886827663"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689886827663"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689886827663"}]},"ts":"1689886827663"} 2023-07-20 21:00:27,670 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 2587dac91101456d664c734eee8a5428, server=jenkins-hbase16.apache.org,34051,1689886824509}] 2023-07-20 21:00:27,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 21:00:27,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:27,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2587dac91101456d664c734eee8a5428, NAME => 't,,1689886827417.2587dac91101456d664c734eee8a5428.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:27,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated t,,1689886827417.2587dac91101456d664c734eee8a5428.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:27,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,842 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,844 DEBUG [StoreOpener-2587dac91101456d664c734eee8a5428-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f 2023-07-20 21:00:27,844 DEBUG [StoreOpener-2587dac91101456d664c734eee8a5428-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f 2023-07-20 21:00:27,845 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2587dac91101456d664c734eee8a5428 columnFamilyName f 2023-07-20 21:00:27,846 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] regionserver.HStore(310): Store=2587dac91101456d664c734eee8a5428/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:27,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:27,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 21:00:27,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 2587dac91101456d664c734eee8a5428; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74257103, jitterRate=0.10651706159114838}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 21:00:27,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 2587dac91101456d664c734eee8a5428: 2023-07-20 21:00:27,856 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689886827417.2587dac91101456d664c734eee8a5428., pid=11, masterSystemTime=1689886827826 2023-07-20 21:00:27,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:27,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:27,860 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=2587dac91101456d664c734eee8a5428, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:27,860 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689886827417.2587dac91101456d664c734eee8a5428.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689886827860"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689886827860"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689886827860"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689886827860"}]},"ts":"1689886827860"} 2023-07-20 21:00:27,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-20 21:00:27,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 2587dac91101456d664c734eee8a5428, server=jenkins-hbase16.apache.org,34051,1689886824509 in 193 msec 2023-07-20 21:00:27,869 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-20 21:00:27,869 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN in 361 msec 2023-07-20 21:00:27,870 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-20 21:00:27,871 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689886827870"}]},"ts":"1689886827870"} 2023-07-20 21:00:27,873 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-20 21:00:27,913 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-20 21:00:27,918 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 492 msec 2023-07-20 21:00:28,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 21:00:28,058 INFO [Listener at localhost.localdomain/46531] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-20 21:00:28,058 DEBUG [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-20 21:00:28,060 INFO [Listener at localhost.localdomain/46531] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-20 21:00:28,065 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-20 21:00:28,066 INFO [Listener at localhost.localdomain/46531] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-20 21:00:28,066 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-20 21:00:28,114 INFO [Listener at localhost.localdomain/46531] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=293, OpenFileDescriptor=581, MaxFileDescriptor=60000, SystemLoadAverage=144, ProcessCount=169, AvailableMemoryMB=8899 2023-07-20 21:00:28,116 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-20 21:00:28,120 DEBUG [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 21:00:28,121 DEBUG [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:28,121 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-20 21:00:28,121 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-20 21:00:28,124 INFO [Listener at localhost.localdomain/46531] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689886827104/Put/vlen=2/seqid=0} 2023-07-20 21:00:28,124 INFO [Listener at localhost.localdomain/46531] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-20 21:00:28,124 INFO [Listener at localhost.localdomain/46531] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b./info:regioninfo/1689886827088/Put/vlen=41/seqid=0, hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b./info:seqnumDuringOpen/1689886827088/Put/vlen=8/seqid=0, hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b./info:server/1689886827088/Put/vlen=32/seqid=0, hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b./info:serverstartcode/1689886827088/Put/vlen=8/seqid=0, hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b./info:sn/1689886826888/Put/vlen=46/seqid=0, hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b./info:state/1689886827088/Put/vlen=4/seqid=0} 2023-07-20 21:00:28,125 INFO [Listener at localhost.localdomain/46531] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => e6b0db802b49804f36d630fc7a235f8b, NAME => 'hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:28,125 INFO [Listener at localhost.localdomain/46531] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689886827870/Put/vlen=2/seqid=0} 2023-07-20 21:00:28,125 INFO [Listener at localhost.localdomain/46531] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-20 21:00:28,125 INFO [Listener at localhost.localdomain/46531] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689886827417.2587dac91101456d664c734eee8a5428./info:regioninfo/1689886827860/Put/vlen=35/seqid=0, t,,1689886827417.2587dac91101456d664c734eee8a5428./info:seqnumDuringOpen/1689886827860/Put/vlen=8/seqid=0, t,,1689886827417.2587dac91101456d664c734eee8a5428./info:server/1689886827860/Put/vlen=32/seqid=0, t,,1689886827417.2587dac91101456d664c734eee8a5428./info:serverstartcode/1689886827860/Put/vlen=8/seqid=0, t,,1689886827417.2587dac91101456d664c734eee8a5428./info:sn/1689886827663/Put/vlen=46/seqid=0, t,,1689886827417.2587dac91101456d664c734eee8a5428./info:state/1689886827860/Put/vlen=4/seqid=0} 2023-07-20 21:00:28,125 INFO [Listener at localhost.localdomain/46531] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 2587dac91101456d664c734eee8a5428, NAME => 't,,1689886827417.2587dac91101456d664c734eee8a5428.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:28,126 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(139): Connect 0x3198ba45 to 127.0.0.1:57031 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 21:00:28,136 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38dd297e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 21:00:28,137 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-20 21:00:28,137 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-20 21:00:28,140 DEBUG [hconnection-0x55af6ef2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:28,143 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:60756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:28,155 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-20 21:00:28,159 ERROR [Listener at localhost.localdomain/46531] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase16.apache.org,34051,1689886824509: die! ***** 2023-07-20 21:00:28,159 ERROR [Listener at localhost.localdomain/46531] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-20 21:00:28,162 DEBUG [Listener at localhost.localdomain/46531] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-20 21:00:28,165 DEBUG [Listener at localhost.localdomain/46531] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-20 21:00:28,165 DEBUG [Listener at localhost.localdomain/46531] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-20 21:00:28,166 DEBUG [Listener at localhost.localdomain/46531] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-20 21:00:28,166 INFO [Listener at localhost.localdomain/46531] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 902299648, "init": 524288000, "max": 2051014656, "used": 395624648 }, "NonHeapMemoryUsage": { "committed": 113205248, "init": 2555904, "max": -1, "used": 111240864 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase16.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase16.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase16.apache.org" } ] } 2023-07-20 21:00:28,169 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38535] master.MasterRpcServices(612): jenkins-hbase16.apache.org,34051,1689886824509 reported a fatal error: ***** ABORTING region server jenkins-hbase16.apache.org,34051,1689886824509: die! ***** 2023-07-20 21:00:28,172 INFO [Listener at localhost.localdomain/46531] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase16.apache.org,34051,1689886824509' ***** 2023-07-20 21:00:28,172 INFO [Listener at localhost.localdomain/46531] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-20 21:00:28,173 INFO [RS:1;jenkins-hbase16:34051] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 21:00:28,173 INFO [RS:1;jenkins-hbase16:34051] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-20 21:00:28,173 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 21:00:28,173 INFO [RS:1;jenkins-hbase16:34051] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-20 21:00:28,173 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(3305): Received CLOSE for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:28,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34051] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 5434273015544152076 2023-07-20 21:00:28,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34051] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 21:00:28,174 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(3305): Received CLOSE for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:28,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 2587dac91101456d664c734eee8a5428, disabling compactions & flushes 2023-07-20 21:00:28,176 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1141): aborting server jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:28,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34051] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 188.40.62.62:60756 deadline: 1689886888174, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting 2023-07-20 21:00:28,176 DEBUG [RS:1;jenkins-hbase16:34051] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28fd7dde to 127.0.0.1:57031 2023-07-20 21:00:28,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:28,176 DEBUG [RS:1;jenkins-hbase16:34051] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:28,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:28,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689886827417.2587dac91101456d664c734eee8a5428. after waiting 0 ms 2023-07-20 21:00:28,176 INFO [RS:1;jenkins-hbase16:34051] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 21:00:28,176 INFO [RS:1;jenkins-hbase16:34051] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 21:00:28,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:28,176 INFO [RS:1;jenkins-hbase16:34051] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 21:00:28,176 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 21:00:28,177 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:28,177 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 21:00:28,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:28,178 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1478): Online Regions={2587dac91101456d664c734eee8a5428=t,,1689886827417.2587dac91101456d664c734eee8a5428., 1588230740=hbase:meta,,1.1588230740, e6b0db802b49804f36d630fc7a235f8b=hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.} 2023-07-20 21:00:28,178 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 21:00:28,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 2587dac91101456d664c734eee8a5428: 2023-07-20 21:00:28,178 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 21:00:28,179 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 21:00:28,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:28,179 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 21:00:28,179 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 21:00:28,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing e6b0db802b49804f36d630fc7a235f8b, disabling compactions & flushes 2023-07-20 21:00:28,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:28,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:28,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. after waiting 0 ms 2023-07-20 21:00:28,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:28,180 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1825): Memstore data size is 2398 in region hbase:meta,,1.1588230740 2023-07-20 21:00:28,180 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 21:00:28,180 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:28,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:28,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for e6b0db802b49804f36d630fc7a235f8b: 2023-07-20 21:00:28,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:28,181 DEBUG [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-20 21:00:28,181 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 21:00:28,181 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 21:00:28,182 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 21:00:28,194 DEBUG [Listener at localhost.localdomain/46531] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-20 21:00:28,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34051] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 21:00:28,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34051] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 188.40.62.62:60756 deadline: 1689886888197, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting 2023-07-20 21:00:28,302 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34051] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 21:00:28,303 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34051] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 188.40.62.62:60756 deadline: 1689886888302, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting 2023-07-20 21:00:28,382 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1170): stopping server jenkins-hbase16.apache.org,34051,1689886824509; all regions closed. 2023-07-20 21:00:28,404 DEBUG [RS:1;jenkins-hbase16:34051] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:28,404 INFO [RS:1;jenkins-hbase16:34051] regionserver.LeaseManager(133): Closed leases 2023-07-20 21:00:28,405 INFO [RS:1;jenkins-hbase16:34051] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-20 21:00:28,405 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 21:00:28,405 INFO [RS:1;jenkins-hbase16:34051] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:34051 2023-07-20 21:00:28,408 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34051] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 21:00:28,408 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34051] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Get size: 76 connection: 188.40.62.62:60756 deadline: 1689886888408, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,34051,1689886824509 aborting 2023-07-20 21:00:28,423 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:28,423 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:28,423 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:28,423 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:28,423 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:28,432 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,34051,1689886824509] 2023-07-20 21:00:28,432 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,34051,1689886824509; numProcessing=1 2023-07-20 21:00:28,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:28,433 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase16.apache.org,34051,1689886824509 znode expired, triggering replicatorRemoved event 2023-07-20 21:00:28,439 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,34051,1689886824509 already deleted, retry=false 2023-07-20 21:00:28,439 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase16.apache.org,34051,1689886824509 on jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:28,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:28,449 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase16.apache.org,34051,1689886824509, splitWal=true, meta=true 2023-07-20 21:00:28,449 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase16.apache.org,34051,1689886824509 (carryingMeta=true) jenkins-hbase16.apache.org,34051,1689886824509/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@325758e2[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-20 21:00:28,453 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase16.apache.org,34051,1689886824509, splitWal=true, meta=true 2023-07-20 21:00:28,454 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase16.apache.org,34051,1689886824509, splitWal=true, meta=true, isMeta: true 2023-07-20 21:00:28,457 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting 2023-07-20 21:00:28,459 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase16.apache.org,34051,1689886824509 WAL count=1, meta=true 2023-07-20 21:00:28,464 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta}] 2023-07-20 21:00:28,473 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:28,474 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta, worker=jenkins-hbase16.apache.org,32921,1689886824455}] 2023-07-20 21:00:28,532 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 21:00:28,533 INFO [RS:1;jenkins-hbase16:34051] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase16.apache.org,34051,1689886824509; zookeeper connection closed. 2023-07-20 21:00:28,533 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:34051-0x1018488ceeb0002, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 21:00:28,534 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@27bfaaff] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@27bfaaff 2023-07-20 21:00:28,645 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase16.apache.org/188.40.62.62:34051 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:34051 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 21:00:28,647 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase16.apache.org/188.40.62.62:34051 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:34051 2023-07-20 21:00:28,651 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:28,651 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 21:00:28,654 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:41780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 21:00:28,655 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=32921] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-20 21:00:28,675 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta, size=3.1 K (3190bytes) 2023-07-20 21:00:28,676 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta 2023-07-20 21:00:28,680 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta after 4ms 2023-07-20 21:00:28,684 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta took 9ms 2023-07-20 21:00:28,697 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-20 21:00:28,700 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta so closing down 2023-07-20 21:00:28,700 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 21:00:28,704 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-20 21:00:28,706 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta.temp 2023-07-20 21:00:28,708 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 21:00:28,721 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-20 21:00:28,723 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta.temp to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 21:00:28,724 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta, size=3.1 K, length=3190, corrupted=false, cancelled=false 2023-07-20 21:00:28,724 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta, journal: Splitting hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta, size=3.1 K (3190bytes) at 1689886828676Finishing writing output for hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta so closing down at 1689886828700 (+24 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta.temp at 1689886828707 (+7 ms)3 split writer threads finished at 1689886828708 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689886828721 (+13 ms)Rename recovered edits hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta.temp to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689886828723 (+2 ms)Processed 11 edits across 1 Regions in 39 ms; skipped=1; WAL=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta, size=3.1 K, length=3190, corrupted=false, cancelled=false at 1689886828724 (+1 ms) 2023-07-20 21:00:28,725 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta 2023-07-20 21:00:28,727 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-20 21:00:28,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.HMaster(3641): Remote procedure done, pid=14 2023-07-20 21:00:28,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs 2023-07-20 21:00:28,742 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-20 21:00:28,743 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta, worker=jenkins-hbase16.apache.org,32921,1689886824455 in 265 msec 2023-07-20 21:00:28,744 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:28,750 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-20 21:00:28,750 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase16.apache.org%2C34051%2C1689886824509.meta.1689886826238.meta, worker=jenkins-hbase16.apache.org,32921,1689886824455 in 284 msec 2023-07-20 21:00:28,752 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting dir is empty, no logs to split. 2023-07-20 21:00:28,752 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase16.apache.org,34051,1689886824509 WAL count=0, meta=true 2023-07-20 21:00:28,752 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase16.apache.org,34051,1689886824509 WAL splitting is done? wals=0, meta=true 2023-07-20 21:00:28,754 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 21:00:28,755 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 21:00:28,756 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 21:00:28,906 DEBUG [jenkins-hbase16:38535] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 21:00:28,907 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,32921,1689886824455, state=OPENING 2023-07-20 21:00:28,944 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 21:00:28,944 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,32921,1689886824455}] 2023-07-20 21:00:28,944 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 21:00:28,951 WARN [ReadOnlyZKClient-127.0.0.1:57031@0x3198ba45] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-20 21:00:28,952 DEBUG [hconnection-0x55af6ef2-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:28,957 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:41784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:28,957 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=32921] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 188.40.62.62:41784 deadline: 1689886888957, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:28,959 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 21:00:29,112 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 21:00:29,112 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 21:00:29,116 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455, archiveDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs, maxLogs=32 2023-07-20 21:00:29,138 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK] 2023-07-20 21:00:29,138 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK] 2023-07-20 21:00:29,142 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta 2023-07-20 21:00:29,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK], DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK]] 2023-07-20 21:00:29,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:29,143 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 21:00:29,143 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 21:00:29,144 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 21:00:29,144 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 21:00:29,144 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:29,144 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 21:00:29,144 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 21:00:29,146 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 21:00:29,148 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info 2023-07-20 21:00:29,148 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info 2023-07-20 21:00:29,149 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 21:00:29,149 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:29,149 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 21:00:29,151 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/rep_barrier 2023-07-20 21:00:29,151 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/rep_barrier 2023-07-20 21:00:29,151 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 21:00:29,152 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:29,152 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 21:00:29,153 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table 2023-07-20 21:00:29,153 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table 2023-07-20 21:00:29,154 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 21:00:29,154 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:29,156 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740 2023-07-20 21:00:29,159 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740 2023-07-20 21:00:29,160 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 21:00:29,168 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 21:00:29,168 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-20 21:00:29,214 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.19 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/.tmp/info/273e181fe7564cae80c2b57bc9cbe14b 2023-07-20 21:00:29,264 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/.tmp/table/97669b3820364653aa58d877176de04f 2023-07-20 21:00:29,273 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/.tmp/info/273e181fe7564cae80c2b57bc9cbe14b as hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info/273e181fe7564cae80c2b57bc9cbe14b 2023-07-20 21:00:29,283 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info/273e181fe7564cae80c2b57bc9cbe14b, entries=20, sequenceid=13, filesize=6.9 K 2023-07-20 21:00:29,285 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/.tmp/table/97669b3820364653aa58d877176de04f as hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table/97669b3820364653aa58d877176de04f 2023-07-20 21:00:29,294 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table/97669b3820364653aa58d877176de04f, entries=4, sequenceid=13, filesize=4.7 K 2023-07-20 21:00:29,294 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2398, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 126ms, sequenceid=13, compaction requested=false; wal=null 2023-07-20 21:00:29,295 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-20 21:00:29,298 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 21:00:29,301 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 21:00:29,302 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 21:00:29,305 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-20 21:00:29,306 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69991869, jitterRate=0.04296012222766876}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 21:00:29,306 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 21:00:29,307 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689886829100 2023-07-20 21:00:29,309 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 21:00:29,310 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 21:00:29,310 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,32921,1689886824455, state=OPEN 2023-07-20 21:00:29,344 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 21:00:29,344 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 21:00:29,348 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-20 21:00:29,348 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,32921,1689886824455 in 400 msec 2023-07-20 21:00:29,352 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-20 21:00:29,352 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 594 msec 2023-07-20 21:00:29,353 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase16.apache.org,34051,1689886824509 had 3 regions 2023-07-20 21:00:29,355 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase16.apache.org,34051,1689886824509, splitWal=true, meta=true, isMeta: false 2023-07-20 21:00:29,359 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase16.apache.org,34051,1689886824509 WAL count=1, meta=false 2023-07-20 21:00:29,359 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115}] 2023-07-20 21:00:29,361 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:29,363 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115, worker=jenkins-hbase16.apache.org,32921,1689886824455}] 2023-07-20 21:00:29,482 DEBUG [hconnection-0x55af6ef2-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase16.apache.org/188.40.62.62:34051 this server is in the failed servers list 2023-07-20 21:00:29,516 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=32921] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-20 21:00:29,532 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115, size=1.6 K (1667bytes) 2023-07-20 21:00:29,532 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115 2023-07-20 21:00:29,533 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115 after 1ms 2023-07-20 21:00:29,536 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(311): Open hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115 took 4ms 2023-07-20 21:00:29,539 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(346): Last flushed sequenceid for e6b0db802b49804f36d630fc7a235f8b: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-20 21:00:29,542 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(346): Last flushed sequenceid for 2587dac91101456d664c734eee8a5428: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-20 21:00:29,543 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115 so closing down 2023-07-20 21:00:29,543 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 21:00:29,544 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp 2023-07-20 21:00:29,546 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp 2023-07-20 21:00:29,547 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 21:00:29,564 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-20 21:00:29,565 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-20 21:00:29,566 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000005 2023-07-20 21:00:29,567 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000013 2023-07-20 21:00:29,567 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 31 ms; skipped=2; WAL=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115, size=1.6 K, length=1667, corrupted=false, cancelled=false 2023-07-20 21:00:29,568 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115, journal: Splitting hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115, size=1.6 K (1667bytes) at 1689886829532Finishing writing output for hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115 so closing down at 1689886829543 (+11 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp at 1689886829544 (+1 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp at 1689886829546 (+2 ms)3 split writer threads finished at 1689886829547 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689886829564 (+17 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689886829565 (+1 ms)Rename recovered edits hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000005 at 1689886829566 (+1 ms)Rename recovered edits hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000004-jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115.temp to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000013 at 1689886829567 (+1 ms)Processed 14 edits across 2 Regions in 31 ms; skipped=2; WAL=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115, size=1.6 K, length=1667, corrupted=false, cancelled=false at 1689886829568 (+1 ms) 2023-07-20 21:00:29,568 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115 2023-07-20 21:00:29,568 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-20 21:00:29,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.HMaster(3641): Remote procedure done, pid=18 2023-07-20 21:00:29,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting/jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115 to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs 2023-07-20 21:00:29,576 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-20 21:00:29,576 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115, worker=jenkins-hbase16.apache.org,32921,1689886824455 in 211 msec 2023-07-20 21:00:29,577 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:29,580 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-20 21:00:29,580 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase16.apache.org%2C34051%2C1689886824509.1689886826115, worker=jenkins-hbase16.apache.org,32921,1689886824455 in 218 msec 2023-07-20 21:00:29,582 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,34051,1689886824509-splitting dir is empty, no logs to split. 2023-07-20 21:00:29,582 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase16.apache.org,34051,1689886824509 WAL count=0, meta=false 2023-07-20 21:00:29,582 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase16.apache.org,34051,1689886824509 WAL splitting is done? wals=0, meta=false 2023-07-20 21:00:29,584 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN}, {pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN}] 2023-07-20 21:00:29,586 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN 2023-07-20 21:00:29,586 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN 2023-07-20 21:00:29,587 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 21:00:29,587 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 21:00:29,587 DEBUG [jenkins-hbase16:38535] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 21:00:29,588 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=2587dac91101456d664c734eee8a5428, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:29,588 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=e6b0db802b49804f36d630fc7a235f8b, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:29,589 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689886829588"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689886829588"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689886829588"}]},"ts":"1689886829588"} 2023-07-20 21:00:29,589 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689886827417.2587dac91101456d664c734eee8a5428.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689886829588"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689886829588"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689886829588"}]},"ts":"1689886829588"} 2023-07-20 21:00:29,589 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase16.apache.org/188.40.62.62:34051 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:34051 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 21:00:29,591 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase16.apache.org/188.40.62.62:34051 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:34051 2023-07-20 21:00:29,693 DEBUG [hconnection-0x55af6ef2-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase16.apache.org/188.40.62.62:34051 this server is in the failed servers list 2023-07-20 21:00:29,694 DEBUG [PEWorker-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:29,697 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:41800, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:29,699 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure e6b0db802b49804f36d630fc7a235f8b, server=jenkins-hbase16.apache.org,32921,1689886824455}] 2023-07-20 21:00:29,700 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE; OpenRegionProcedure 2587dac91101456d664c734eee8a5428, server=jenkins-hbase16.apache.org,32921,1689886824455}] 2023-07-20 21:00:29,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:29,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2587dac91101456d664c734eee8a5428, NAME => 't,,1689886827417.2587dac91101456d664c734eee8a5428.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:29,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:29,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated t,,1689886827417.2587dac91101456d664c734eee8a5428.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:29,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:29,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:29,866 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:29,868 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:29,868 DEBUG [StoreOpener-2587dac91101456d664c734eee8a5428-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f 2023-07-20 21:00:29,868 DEBUG [StoreOpener-2587dac91101456d664c734eee8a5428-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f 2023-07-20 21:00:29,868 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e6b0db802b49804f36d630fc7a235f8b, NAME => 'hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:29,869 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:29,869 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2587dac91101456d664c734eee8a5428 columnFamilyName f 2023-07-20 21:00:29,869 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:29,869 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:29,869 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:29,869 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] regionserver.HStore(310): Store=2587dac91101456d664c734eee8a5428/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:29,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428 2023-07-20 21:00:29,873 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:29,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428 2023-07-20 21:00:29,874 DEBUG [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info 2023-07-20 21:00:29,875 DEBUG [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info 2023-07-20 21:00:29,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000013 2023-07-20 21:00:29,875 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6b0db802b49804f36d630fc7a235f8b columnFamilyName info 2023-07-20 21:00:29,876 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] regionserver.HStore(310): Store=e6b0db802b49804f36d630fc7a235f8b/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:29,878 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:29,881 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:29,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000013 2023-07-20 21:00:29,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 2587dac91101456d664c734eee8a5428 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-20 21:00:29,882 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000005 2023-07-20 21:00:29,888 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000005 2023-07-20 21:00:29,888 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing e6b0db802b49804f36d630fc7a235f8b 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-20 21:00:29,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/.tmp/f/863335799b7b4a6ba9d31e57a9b6e34d 2023-07-20 21:00:29,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/.tmp/f/863335799b7b4a6ba9d31e57a9b6e34d as hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f/863335799b7b4a6ba9d31e57a9b6e34d 2023-07-20 21:00:29,917 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/.tmp/info/7105be2420954d3fb1f40ae1d0a97f75 2023-07-20 21:00:29,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f/863335799b7b4a6ba9d31e57a9b6e34d, entries=10, sequenceid=13, filesize=4.8 K 2023-07-20 21:00:29,924 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 2587dac91101456d664c734eee8a5428 in 42ms, sequenceid=13, compaction requested=false; wal=null 2023-07-20 21:00:29,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-20 21:00:29,925 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/.tmp/info/7105be2420954d3fb1f40ae1d0a97f75 as hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info/7105be2420954d3fb1f40ae1d0a97f75 2023-07-20 21:00:29,926 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/0000000000000000013 2023-07-20 21:00:29,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:29,933 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info/7105be2420954d3fb1f40ae1d0a97f75, entries=2, sequenceid=5, filesize=4.8 K 2023-07-20 21:00:29,935 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e6b0db802b49804f36d630fc7a235f8b in 47ms, sequenceid=5, compaction requested=false; wal=null 2023-07-20 21:00:29,935 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-20 21:00:29,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-20 21:00:29,937 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/0000000000000000005 2023-07-20 21:00:29,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 2587dac91101456d664c734eee8a5428; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71850331, jitterRate=0.07065336406230927}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 21:00:29,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 2587dac91101456d664c734eee8a5428: 2023-07-20 21:00:29,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689886827417.2587dac91101456d664c734eee8a5428., pid=22, masterSystemTime=1689886829851 2023-07-20 21:00:29,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:29,941 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:29,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:29,942 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=2587dac91101456d664c734eee8a5428, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:29,942 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689886827417.2587dac91101456d664c734eee8a5428.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689886829942"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689886829942"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689886829942"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689886829942"}]},"ts":"1689886829942"} 2023-07-20 21:00:29,944 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-20 21:00:29,945 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened e6b0db802b49804f36d630fc7a235f8b; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71439212, jitterRate=0.06452721357345581}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 21:00:29,945 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for e6b0db802b49804f36d630fc7a235f8b: 2023-07-20 21:00:29,946 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b., pid=21, masterSystemTime=1689886829851 2023-07-20 21:00:29,948 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-07-20 21:00:29,948 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; OpenRegionProcedure 2587dac91101456d664c734eee8a5428, server=jenkins-hbase16.apache.org,32921,1689886824455 in 245 msec 2023-07-20 21:00:29,948 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:29,948 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:29,949 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=e6b0db802b49804f36d630fc7a235f8b, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:29,950 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689886829949"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689886829949"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689886829949"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689886829949"}]},"ts":"1689886829949"} 2023-07-20 21:00:29,950 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN in 364 msec 2023-07-20 21:00:29,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-20 21:00:29,955 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure e6b0db802b49804f36d630fc7a235f8b, server=jenkins-hbase16.apache.org,32921,1689886824455 in 253 msec 2023-07-20 21:00:29,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=12 2023-07-20 21:00:29,958 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase16.apache.org,34051,1689886824509 after splitting done 2023-07-20 21:00:29,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN in 371 msec 2023-07-20 21:00:29,958 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase16.apache.org,34051,1689886824509 from processing; numProcessing=0 2023-07-20 21:00:29,961 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase16.apache.org,34051,1689886824509, splitWal=true, meta=true in 1.5170 sec 2023-07-20 21:00:30,010 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3198ba45 to 127.0.0.1:57031 2023-07-20 21:00:30,011 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:30,011 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-20 21:00:30,023 INFO [Listener at localhost.localdomain/46531] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=272 (was 293), OpenFileDescriptor=580 (was 581), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=144 (was 144), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=8839 (was 8899) 2023-07-20 21:00:30,035 INFO [Listener at localhost.localdomain/46531] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=272, OpenFileDescriptor=580, MaxFileDescriptor=60000, SystemLoadAverage=144, ProcessCount=170, AvailableMemoryMB=8838 2023-07-20 21:00:30,035 INFO [Listener at localhost.localdomain/46531] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase16:34051 2023-07-20 21:00:30,049 INFO [Listener at localhost.localdomain/46531] client.ConnectionUtils(127): regionserver/jenkins-hbase16:0 server-side Connection retries=45 2023-07-20 21:00:30,049 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:30,049 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:30,050 INFO [Listener at localhost.localdomain/46531] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 21:00:30,050 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 21:00:30,050 INFO [Listener at localhost.localdomain/46531] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 21:00:30,050 INFO [Listener at localhost.localdomain/46531] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 21:00:30,052 INFO [Listener at localhost.localdomain/46531] ipc.NettyRpcServer(120): Bind to /188.40.62.62:36285 2023-07-20 21:00:30,052 INFO [Listener at localhost.localdomain/46531] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 21:00:30,054 DEBUG [Listener at localhost.localdomain/46531] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 21:00:30,054 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:30,056 INFO [Listener at localhost.localdomain/46531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 21:00:30,057 INFO [Listener at localhost.localdomain/46531] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36285 connecting to ZooKeeper ensemble=127.0.0.1:57031 2023-07-20 21:00:30,069 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:362850x0, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 21:00:30,070 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(162): regionserver:362850x0, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 21:00:30,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36285-0x1018488ceeb0008 connected 2023-07-20 21:00:30,072 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(162): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-20 21:00:30,073 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ZKUtil(164): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 21:00:30,073 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36285 2023-07-20 21:00:30,073 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36285 2023-07-20 21:00:30,074 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36285 2023-07-20 21:00:30,074 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36285 2023-07-20 21:00:30,074 DEBUG [Listener at localhost.localdomain/46531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36285 2023-07-20 21:00:30,077 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(951): ClusterId : dbbe0cde-59ef-4799-85ef-0469c06fee88 2023-07-20 21:00:30,078 DEBUG [RS:2;jenkins-hbase16:36285] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 21:00:30,086 DEBUG [RS:2;jenkins-hbase16:36285] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 21:00:30,086 DEBUG [RS:2;jenkins-hbase16:36285] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 21:00:30,095 DEBUG [RS:2;jenkins-hbase16:36285] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 21:00:30,096 DEBUG [RS:2;jenkins-hbase16:36285] zookeeper.ReadOnlyZKClient(139): Connect 0x2aa7af8b to 127.0.0.1:57031 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 21:00:30,112 DEBUG [RS:2;jenkins-hbase16:36285] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@246054af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 21:00:30,113 DEBUG [RS:2;jenkins-hbase16:36285] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@205fc4fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-07-20 21:00:30,126 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase16:36285 2023-07-20 21:00:30,126 INFO [RS:2;jenkins-hbase16:36285] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 21:00:30,126 INFO [RS:2;jenkins-hbase16:36285] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 21:00:30,126 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 21:00:30,126 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase16.apache.org,38535,1689886823355 with isa=jenkins-hbase16.apache.org/188.40.62.62:36285, startcode=1689886830049 2023-07-20 21:00:30,126 DEBUG [RS:2;jenkins-hbase16:36285] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 21:00:30,130 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:53013, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 21:00:30,130 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38535] master.ServerManager(394): Registering regionserver=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:30,131 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce 2023-07-20 21:00:30,131 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:36541 2023-07-20 21:00:30,131 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 21:00:30,139 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:30,139 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:30,140 DEBUG [RS:2;jenkins-hbase16:36285] zookeeper.ZKUtil(162): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:30,140 WARN [RS:2;jenkins-hbase16:36285] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 21:00:30,140 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase16.apache.org,36285,1689886830049] 2023-07-20 21:00:30,140 INFO [RS:2;jenkins-hbase16:36285] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 21:00:30,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:30,140 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:30,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:30,146 DEBUG [RS:2;jenkins-hbase16:36285] zookeeper.ZKUtil(162): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:30,146 DEBUG [RS:2;jenkins-hbase16:36285] zookeeper.ZKUtil(162): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:30,148 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 21:00:30,148 INFO [RS:2;jenkins-hbase16:36285] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 21:00:30,150 INFO [RS:2;jenkins-hbase16:36285] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 21:00:30,150 INFO [RS:2;jenkins-hbase16:36285] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 21:00:30,150 INFO [RS:2;jenkins-hbase16:36285] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:30,151 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 21:00:30,152 INFO [RS:2;jenkins-hbase16:36285] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:30,152 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,152 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,152 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,153 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,153 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,153 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0, corePoolSize=2, maxPoolSize=2 2023-07-20 21:00:30,153 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,153 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,153 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,153 DEBUG [RS:2;jenkins-hbase16:36285] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase16:0, corePoolSize=1, maxPoolSize=1 2023-07-20 21:00:30,155 INFO [RS:2;jenkins-hbase16:36285] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:30,155 INFO [RS:2;jenkins-hbase16:36285] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:30,155 INFO [RS:2;jenkins-hbase16:36285] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:30,167 INFO [RS:2;jenkins-hbase16:36285] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 21:00:30,167 INFO [RS:2;jenkins-hbase16:36285] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase16.apache.org,36285,1689886830049-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 21:00:30,177 INFO [RS:2;jenkins-hbase16:36285] regionserver.Replication(203): jenkins-hbase16.apache.org,36285,1689886830049 started 2023-07-20 21:00:30,177 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1637): Serving as jenkins-hbase16.apache.org,36285,1689886830049, RpcServer on jenkins-hbase16.apache.org/188.40.62.62:36285, sessionid=0x1018488ceeb0008 2023-07-20 21:00:30,177 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase16:36285,5,FailOnTimeoutGroup] 2023-07-20 21:00:30,177 DEBUG [RS:2;jenkins-hbase16:36285] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 21:00:30,177 DEBUG [RS:2;jenkins-hbase16:36285] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:30,177 DEBUG [RS:2;jenkins-hbase16:36285] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,36285,1689886830049' 2023-07-20 21:00:30,177 DEBUG [RS:2;jenkins-hbase16:36285] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 21:00:30,177 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:30,177 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(177): START ************ test3686b 2023-07-20 21:00:30,178 DEBUG [RS:2;jenkins-hbase16:36285] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 21:00:30,178 DEBUG [RS:2;jenkins-hbase16:36285] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 21:00:30,178 DEBUG [RS:2;jenkins-hbase16:36285] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 21:00:30,178 DEBUG [RS:2;jenkins-hbase16:36285] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:30,179 DEBUG [RS:2;jenkins-hbase16:36285] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase16.apache.org,36285,1689886830049' 2023-07-20 21:00:30,179 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase16.apache.org/188.40.62.62:34051 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:34051 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 21:00:30,179 DEBUG [RS:2;jenkins-hbase16:36285] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 21:00:30,180 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase16.apache.org/188.40.62.62:34051 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:34051 2023-07-20 21:00:30,181 DEBUG [RS:2;jenkins-hbase16:36285] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 21:00:30,181 DEBUG [RS:2;jenkins-hbase16:36285] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 21:00:30,181 INFO [RS:2;jenkins-hbase16:36285] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 21:00:30,181 INFO [RS:2;jenkins-hbase16:36285] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 21:00:30,284 DEBUG [hconnection-0x354830d7-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:30,286 INFO [RS:2;jenkins-hbase16:36285] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C36285%2C1689886830049, suffix=, logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,36285,1689886830049, archiveDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs, maxLogs=32 2023-07-20 21:00:30,289 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:41812, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:30,293 DEBUG [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 21:00:30,293 DEBUG [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:30,294 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(139): Connect 0x278c7448 to 127.0.0.1:57031 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 21:00:30,311 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK] 2023-07-20 21:00:30,316 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK] 2023-07-20 21:00:30,316 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c1767ed, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 21:00:30,321 DEBUG [hconnection-0x31be2331-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:30,322 INFO [RS:2;jenkins-hbase16:36285] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,36285,1689886830049/jenkins-hbase16.apache.org%2C36285%2C1689886830049.1689886830290 2023-07-20 21:00:30,323 DEBUG [RS:2;jenkins-hbase16:36285] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK], DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK]] 2023-07-20 21:00:30,325 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:41818, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:31,845 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-20 21:00:33,506 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 21:00:33,508 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-20 21:00:45,335 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 3398697798458146823 expired clientIPAndPort=188.40.62.62:41818, userName=jenkins, regionInfo=t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:47,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32921] regionserver.RSRpcServices(3094): Client tried to access missing scanner 3398697798458146823 2023-07-20 21:00:47,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32921] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 188.40.62.62:41818 deadline: 1689886907336, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '3398697798458146823'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-20 21:00:47,341 DEBUG [Listener at localhost.localdomain/46531] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '3398697798458146823'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '3398697798458146823'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '3398697798458146823'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-20 21:00:47,347 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x278c7448 to 127.0.0.1:57031 2023-07-20 21:00:47,347 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:47,347 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-20 21:00:47,365 INFO [Listener at localhost.localdomain/46531] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=298 (was 272) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57031@0x278c7448-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce-prefix:jenkins-hbase16.apache.org,36285,1689886830049 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase16:36285Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57031@0x2aa7af8b-SendThread(127.0.0.1:57031) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57031@0x2aa7af8b sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1417955001.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase16:36285-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase16:36285 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57031@0x2aa7af8b-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: PacketResponder: BP-1827106336-188.40.62.62-1689886819716:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost.localdomain:36541 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1234117007_17 at /127.0.0.1:47344 [Receiving block BP-1827106336-188.40.62.62-1689886819716:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1827106336-188.40.62.62-1689886819716:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57031@0x278c7448-SendThread(127.0.0.1:57031) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1234117007_17 at /127.0.0.1:37330 [Receiving block BP-1827106336-188.40.62.62-1689886819716:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57031@0x278c7448 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1417955001.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=36285 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=574 (was 580), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=120 (was 144), ProcessCount=170 (was 170), AvailableMemoryMB=8821 (was 8838) 2023-07-20 21:00:47,379 INFO [Listener at localhost.localdomain/46531] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=298, OpenFileDescriptor=574, MaxFileDescriptor=60000, SystemLoadAverage=120, ProcessCount=170, AvailableMemoryMB=8820 2023-07-20 21:00:47,380 INFO [Listener at localhost.localdomain/46531] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase16:34051 2023-07-20 21:00:47,380 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase16.apache.org,34051,1689886824509 2023-07-20 21:00:47,380 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(104): START************ test2772 2023-07-20 21:00:47,384 DEBUG [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 21:00:47,384 DEBUG [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:47,384 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(139): Connect 0x6dd2374a to 127.0.0.1:57031 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 21:00:47,398 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38dcafec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 21:00:47,398 ERROR [Listener at localhost.localdomain/46531] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase16.apache.org,32921,1689886824455: die! ***** 2023-07-20 21:00:47,398 ERROR [Listener at localhost.localdomain/46531] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-20 21:00:47,398 DEBUG [Listener at localhost.localdomain/46531] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-20 21:00:47,399 DEBUG [Listener at localhost.localdomain/46531] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-20 21:00:47,406 DEBUG [Listener at localhost.localdomain/46531] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-20 21:00:47,409 DEBUG [Listener at localhost.localdomain/46531] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-20 21:00:47,423 INFO [Listener at localhost.localdomain/46531] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 990380032, "init": 524288000, "max": 2051014656, "used": 272669424 }, "NonHeapMemoryUsage": { "committed": 124608512, "init": 2555904, "max": -1, "used": 122132264 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase16.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13382, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 4, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 2, "ProcessCallTime_max": 2, "ProcessCallTime_mean": 2, "ProcessCallTime_25th_percentile": 2, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 2, "ProcessCallTime_95th_percentile": 2, "ProcessCallTime_98th_percentile": 2, "ProcessCallTime_99th_percentile": 2, "ProcessCallTime_99.9th_percentile": 2, "ProcessCallTime_TimeRangeCount_1-3": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 2, "TotalCallTime_max": 2, "TotalCallTime_mean": 2, "TotalCallTime_25th_percentile": 2, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 2, "TotalCallTime_95th_percentile": 2, "TotalCallTime_98th_percentile": 2, "TotalCallTime_99th_percentile": 2, "TotalCallTime_99.9th_percentile": 2, "TotalCallTime_TimeRangeCount_1-3": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 196, "ResponseSize_max": 680, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 197, "ResponseSize_75th_percentile": 677, "ResponseSize_90th_percentile": 679, "ResponseSize_95th_percentile": 679, "ResponseSize_98th_percentile": 679, "ResponseSize_99th_percentile": 679, "ResponseSize_99.9th_percentile": 679, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 6, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16964 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase16.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:57031", "tag.serverName": "jenkins-hbase16.apache.org,34051,1689886824509", "tag.clusterId": "dbbe0cde-59ef-4799-85ef-0469c06fee88", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase16.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689886824509, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2492, "localBytesRead": 2492, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 8754, "ByteBuffAllocatorPoolAllocationBytes": 998400, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 40, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 6, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 12, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 22, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 12, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 22, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 12, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 52, "ServerReadQueryPerSecond_mean_rate": 2.4211576001413344, "ServerReadQueryPerSecond_1min_rate": 7.164967204256925, "ServerReadQueryPerSecond_5min_rate": 8.751310705406569, "ServerReadQueryPerSecond_15min_rate": 9.047937375158883, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.2105788005215905, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-20 21:00:47,423 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38535] master.MasterRpcServices(612): jenkins-hbase16.apache.org,32921,1689886824455 reported a fatal error: ***** ABORTING region server jenkins-hbase16.apache.org,32921,1689886824455: die! ***** 2023-07-20 21:00:47,425 INFO [Listener at localhost.localdomain/46531] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase16.apache.org,32921,1689886824455' ***** 2023-07-20 21:00:47,425 INFO [Listener at localhost.localdomain/46531] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-20 21:00:47,425 INFO [RS:0;jenkins-hbase16:32921] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 21:00:47,425 INFO [RS:0;jenkins-hbase16:32921] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-20 21:00:47,425 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 21:00:47,425 INFO [RS:0;jenkins-hbase16:32921] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-20 21:00:47,426 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(3305): Received CLOSE for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:47,426 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(3305): Received CLOSE for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:47,426 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1141): aborting server jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:47,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 2587dac91101456d664c734eee8a5428, disabling compactions & flushes 2023-07-20 21:00:47,427 DEBUG [RS:0;jenkins-hbase16:32921] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5cd9e5fe to 127.0.0.1:57031 2023-07-20 21:00:47,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:47,427 DEBUG [RS:0;jenkins-hbase16:32921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:47,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:47,427 INFO [RS:0;jenkins-hbase16:32921] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 21:00:47,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689886827417.2587dac91101456d664c734eee8a5428. after waiting 0 ms 2023-07-20 21:00:47,427 INFO [RS:0;jenkins-hbase16:32921] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 21:00:47,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:47,427 INFO [RS:0;jenkins-hbase16:32921] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 21:00:47,427 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 21:00:47,430 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 21:00:47,430 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1478): Online Regions={2587dac91101456d664c734eee8a5428=t,,1689886827417.2587dac91101456d664c734eee8a5428., 1588230740=hbase:meta,,1.1588230740, e6b0db802b49804f36d630fc7a235f8b=hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.} 2023-07-20 21:00:47,431 DEBUG [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1504): Waiting on 1588230740, 2587dac91101456d664c734eee8a5428, e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:47,433 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 21:00:47,434 DEBUG [hconnection-0x71f5e0bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:47,434 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 21:00:47,435 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 21:00:47,435 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 21:00:47,435 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 21:00:47,435 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:47,435 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 2587dac91101456d664c734eee8a5428: 2023-07-20 21:00:47,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:47,437 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55440, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:47,437 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=32921] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,32921,1689886824455 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 21:00:47,438 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=32921] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 188.40.62.62:55440 deadline: 1689886907437, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,32921,1689886824455 aborting 2023-07-20 21:00:47,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing e6b0db802b49804f36d630fc7a235f8b, disabling compactions & flushes 2023-07-20 21:00:47,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:47,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:47,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. after waiting 0 ms 2023-07-20 21:00:47,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:47,441 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1825): Memstore data size is 1796 in region hbase:meta,,1.1588230740 2023-07-20 21:00:47,442 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 21:00:47,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:47,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for e6b0db802b49804f36d630fc7a235f8b: 2023-07-20 21:00:47,442 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 21:00:47,443 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 21:00:47,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:47,443 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 21:00:47,543 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=32921] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,32921,1689886824455 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 21:00:47,543 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=32921] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 188.40.62.62:55440 deadline: 1689886907542, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase16.apache.org,32921,1689886824455 aborting 2023-07-20 21:00:47,580 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 21:00:47,631 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1170): stopping server jenkins-hbase16.apache.org,32921,1689886824455; all regions closed. 2023-07-20 21:00:47,654 DEBUG [RS:0;jenkins-hbase16:32921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:47,654 INFO [RS:0;jenkins-hbase16:32921] regionserver.LeaseManager(133): Closed leases 2023-07-20 21:00:47,654 INFO [RS:0;jenkins-hbase16:32921] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-20 21:00:47,655 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 21:00:47,655 INFO [RS:0;jenkins-hbase16:32921] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:32921 2023-07-20 21:00:47,668 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:47,668 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:47,668 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 2023-07-20 21:00:47,668 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:47,668 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:47,676 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,32921,1689886824455] 2023-07-20 21:00:47,676 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,32921,1689886824455; numProcessing=1 2023-07-20 21:00:47,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:47,677 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase16.apache.org,32921,1689886824455 znode expired, triggering replicatorRemoved event 2023-07-20 21:00:47,685 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,32921,1689886824455 already deleted, retry=false 2023-07-20 21:00:47,685 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase16.apache.org,32921,1689886824455 on jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:47,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:47,686 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=23, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase16.apache.org,32921,1689886824455, splitWal=true, meta=true 2023-07-20 21:00:47,686 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=23 for jenkins-hbase16.apache.org,32921,1689886824455 (carryingMeta=true) jenkins-hbase16.apache.org,32921,1689886824455/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@353cfd40[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-20 21:00:47,688 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=23, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase16.apache.org,32921,1689886824455, splitWal=true, meta=true 2023-07-20 21:00:47,690 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase16.apache.org,32921,1689886824455, splitWal=true, meta=true, isMeta: true 2023-07-20 21:00:47,694 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting 2023-07-20 21:00:47,695 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase16.apache.org,32921,1689886824455 WAL count=1, meta=true 2023-07-20 21:00:47,695 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta}] 2023-07-20 21:00:47,699 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:47,700 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta, worker=jenkins-hbase16.apache.org,36285,1689886830049}] 2023-07-20 21:00:47,747 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase16.apache.org/188.40.62.62:32921 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:32921 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 21:00:47,748 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase16.apache.org/188.40.62.62:32921 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:32921 2023-07-20 21:00:47,776 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 21:00:47,777 INFO [RS:0;jenkins-hbase16:32921] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase16.apache.org,32921,1689886824455; zookeeper connection closed. 2023-07-20 21:00:47,777 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:32921-0x1018488ceeb0001, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 21:00:47,777 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@42632023] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@42632023 2023-07-20 21:00:47,854 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:47,854 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 21:00:47,857 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:55074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 21:00:47,858 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36285] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-20 21:00:47,877 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta, size=2.4 K (2408bytes) 2023-07-20 21:00:47,878 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta 2023-07-20 21:00:47,878 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta after 0ms 2023-07-20 21:00:47,883 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta took 6ms 2023-07-20 21:00:47,887 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-20 21:00:47,888 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta so closing down 2023-07-20 21:00:47,888 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 21:00:47,893 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta.temp 2023-07-20 21:00:47,893 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 21:00:47,902 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) 2023-07-20 21:00:47,904 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta.temp to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-20 21:00:47,904 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(423): Processed 5 edits across 1 Regions in 21 ms; skipped=1; WAL=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta, size=2.4 K, length=2408, corrupted=false, cancelled=false 2023-07-20 21:00:47,905 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta, journal: Splitting hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta, size=2.4 K (2408bytes) at 1689886847878Finishing writing output for hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta so closing down at 1689886847888 (+10 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta.temp at 1689886847893 (+5 ms)3 split writer threads finished at 1689886847893Closed recovered edits writer path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) at 1689886847903 (+10 ms)Rename recovered edits hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta.temp to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 at 1689886847904 (+1 ms)Processed 5 edits across 1 Regions in 21 ms; skipped=1; WAL=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta, size=2.4 K, length=2408, corrupted=false, cancelled=false at 1689886847904 2023-07-20 21:00:47,905 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta 2023-07-20 21:00:47,905 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-20 21:00:47,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.HMaster(3641): Remote procedure done, pid=25 2023-07-20 21:00:47,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs 2023-07-20 21:00:47,912 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-20 21:00:47,912 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta, worker=jenkins-hbase16.apache.org,36285,1689886830049 in 210 msec 2023-07-20 21:00:47,913 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:47,915 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=23 2023-07-20 21:00:47,915 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase16.apache.org%2C32921%2C1689886824455.meta.1689886829118.meta, worker=jenkins-hbase16.apache.org,36285,1689886830049 in 218 msec 2023-07-20 21:00:47,917 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting dir is empty, no logs to split. 2023-07-20 21:00:47,917 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase16.apache.org,32921,1689886824455 WAL count=0, meta=true 2023-07-20 21:00:47,917 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase16.apache.org,32921,1689886824455 WAL splitting is done? wals=0, meta=true 2023-07-20 21:00:47,918 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 21:00:47,919 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 21:00:47,920 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 21:00:48,051 DEBUG [hconnection-0x71f5e0bf-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase16.apache.org/188.40.62.62:32921 this server is in the failed servers list 2023-07-20 21:00:48,070 DEBUG [jenkins-hbase16:38535] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 21:00:48,070 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,36285,1689886830049, state=OPENING 2023-07-20 21:00:48,096 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 21:00:48,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,36285,1689886830049}] 2023-07-20 21:00:48,097 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 21:00:48,260 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 21:00:48,260 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 21:00:48,263 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase16.apache.org%2C36285%2C1689886830049.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,36285,1689886830049, archiveDir=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs, maxLogs=32 2023-07-20 21:00:48,284 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK] 2023-07-20 21:00:48,284 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK] 2023-07-20 21:00:48,286 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,36285,1689886830049/jenkins-hbase16.apache.org%2C36285%2C1689886830049.meta.1689886848265.meta 2023-07-20 21:00:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45727,DS-620eff30-290e-44ba-a990-7757dd03f2c2,DISK], DatanodeInfoWithStorage[127.0.0.1:38951,DS-05acea34-49d1-4d24-8294-9af79e6a70d2,DISK]] 2023-07-20 21:00:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 21:00:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 21:00:48,287 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 21:00:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 21:00:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 21:00:48,287 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 21:00:48,290 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 21:00:48,292 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info 2023-07-20 21:00:48,292 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info 2023-07-20 21:00:48,292 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 21:00:48,304 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info/273e181fe7564cae80c2b57bc9cbe14b 2023-07-20 21:00:48,304 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:48,304 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 21:00:48,306 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/rep_barrier 2023-07-20 21:00:48,306 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/rep_barrier 2023-07-20 21:00:48,307 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 21:00:48,308 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:48,308 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 21:00:48,309 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table 2023-07-20 21:00:48,309 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table 2023-07-20 21:00:48,310 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 21:00:48,320 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/table/97669b3820364653aa58d877176de04f 2023-07-20 21:00:48,320 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:48,322 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740 2023-07-20 21:00:48,324 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740 2023-07-20 21:00:48,325 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-20 21:00:48,329 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5464): Applied 16, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=19, path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-20 21:00:48,329 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-20 21:00:48,347 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=19 (bloomFilter=false), to=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/.tmp/info/5cac7df7aad84b5e81fc4a553f1ed4cf 2023-07-20 21:00:48,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/.tmp/info/5cac7df7aad84b5e81fc4a553f1ed4cf as hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info/5cac7df7aad84b5e81fc4a553f1ed4cf 2023-07-20 21:00:48,364 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info/5cac7df7aad84b5e81fc4a553f1ed4cf, entries=16, sequenceid=19, filesize=6.4 K 2023-07-20 21:00:48,364 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1796, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 35ms, sequenceid=19, compaction requested=false; wal=null 2023-07-20 21:00:48,365 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-20 21:00:48,367 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 21:00:48,369 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 21:00:48,371 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=13 2023-07-20 21:00:48,372 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=20; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62743736, jitterRate=-0.06504547595977783}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 21:00:48,372 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 21:00:48,373 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=27, masterSystemTime=1689886848253 2023-07-20 21:00:48,375 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 21:00:48,376 INFO [RS_OPEN_META-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 21:00:48,377 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase16.apache.org,36285,1689886830049, state=OPEN 2023-07-20 21:00:48,384 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 21:00:48,385 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 21:00:48,387 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-20 21:00:48,388 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase16.apache.org,36285,1689886830049 in 288 msec 2023-07-20 21:00:48,390 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=23 2023-07-20 21:00:48,390 INFO [PEWorker-1] procedure.ServerCrashProcedure(199): jenkins-hbase16.apache.org,32921,1689886824455 had 3 regions 2023-07-20 21:00:48,390 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 469 msec 2023-07-20 21:00:48,391 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase16.apache.org,32921,1689886824455, splitWal=true, meta=true, isMeta: false 2023-07-20 21:00:48,393 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase16.apache.org,32921,1689886824455 WAL count=1, meta=false 2023-07-20 21:00:48,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115}] 2023-07-20 21:00:48,395 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:48,396 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115, worker=jenkins-hbase16.apache.org,36285,1689886830049}] 2023-07-20 21:00:48,549 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36285] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=29 2023-07-20 21:00:48,557 DEBUG [hconnection-0x71f5e0bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:48,560 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:56590, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:48,567 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115, size=723 (723bytes) 2023-07-20 21:00:48,567 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115 2023-07-20 21:00:48,568 DEBUG [hconnection-0x71f5e0bf-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase16.apache.org/188.40.62.62:32921 this server is in the failed servers list 2023-07-20 21:00:48,568 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115 after 1ms 2023-07-20 21:00:48,571 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(311): Open hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115 took 4ms 2023-07-20 21:00:48,575 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(346): Last flushed sequenceid for 2587dac91101456d664c734eee8a5428: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-20 21:00:48,576 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(346): Last flushed sequenceid for e6b0db802b49804f36d630fc7a235f8b: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-20 21:00:48,576 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115 so closing down 2023-07-20 21:00:48,576 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 21:00:48,577 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 21:00:48,577 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 6 ms; skipped=2; WAL=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115, size=723, length=723, corrupted=false, cancelled=false 2023-07-20 21:00:48,577 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115, journal: Splitting hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115, size=723 (723bytes) at 1689886848567Finishing writing output for hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115 so closing down at 1689886848576 (+9 ms)3 split writer threads finished at 1689886848577 (+1 ms)Processed 2 edits across 0 Regions in 6 ms; skipped=2; WAL=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115, size=723, length=723, corrupted=false, cancelled=false at 1689886848577 2023-07-20 21:00:48,577 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115 2023-07-20 21:00:48,577 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase16:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=29 2023-07-20 21:00:48,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] master.HMaster(3641): Remote procedure done, pid=29 2023-07-20 21:00:48,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38535] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting/jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115 to hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs 2023-07-20 21:00:48,584 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-07-20 21:00:48,584 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115, worker=jenkins-hbase16.apache.org,36285,1689886830049 in 186 msec 2023-07-20 21:00:48,585 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:48,588 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=23 2023-07-20 21:00:48,588 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase16.apache.org%2C32921%2C1689886824455.1689886826115, worker=jenkins-hbase16.apache.org,36285,1689886830049 in 192 msec 2023-07-20 21:00:48,590 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/WALs/jenkins-hbase16.apache.org,32921,1689886824455-splitting dir is empty, no logs to split. 2023-07-20 21:00:48,590 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase16.apache.org,32921,1689886824455 WAL count=0, meta=false 2023-07-20 21:00:48,590 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase16.apache.org,32921,1689886824455 WAL splitting is done? wals=0, meta=false 2023-07-20 21:00:48,592 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN}, {pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN}] 2023-07-20 21:00:48,593 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN 2023-07-20 21:00:48,593 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN 2023-07-20 21:00:48,594 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 21:00:48,594 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 21:00:48,594 DEBUG [jenkins-hbase16:38535] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 21:00:48,595 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=e6b0db802b49804f36d630fc7a235f8b, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:48,595 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=2587dac91101456d664c734eee8a5428, regionState=OPENING, regionLocation=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:48,595 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689886827417.2587dac91101456d664c734eee8a5428.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689886848595"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689886848595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689886848595"}]},"ts":"1689886848595"} 2023-07-20 21:00:48,595 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689886848595"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689886848595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689886848595"}]},"ts":"1689886848595"} 2023-07-20 21:00:48,596 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase16.apache.org/188.40.62.62:32921 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:32921 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 21:00:48,596 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase16.apache.org/188.40.62.62:32921 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase16.apache.org/188.40.62.62:32921 2023-07-20 21:00:48,683 DEBUG [hconnection-0x71f5e0bf-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase16.apache.org/188.40.62.62:32921 this server is in the failed servers list 2023-07-20 21:00:48,699 DEBUG [PEWorker-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 21:00:48,701 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 188.40.62.62:56604, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 21:00:48,704 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=30, state=RUNNABLE; OpenRegionProcedure 2587dac91101456d664c734eee8a5428, server=jenkins-hbase16.apache.org,36285,1689886830049}] 2023-07-20 21:00:48,705 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure e6b0db802b49804f36d630fc7a235f8b, server=jenkins-hbase16.apache.org,36285,1689886830049}] 2023-07-20 21:00:48,863 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:48,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e6b0db802b49804f36d630fc7a235f8b, NAME => 'hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:48,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:48,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:48,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:48,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:48,865 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:48,866 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(130): Open t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:48,866 DEBUG [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info 2023-07-20 21:00:48,866 DEBUG [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info 2023-07-20 21:00:48,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2587dac91101456d664c734eee8a5428, NAME => 't,,1689886827417.2587dac91101456d664c734eee8a5428.', STARTKEY => '', ENDKEY => ''} 2023-07-20 21:00:48,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:48,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(866): Instantiated t,,1689886827417.2587dac91101456d664c734eee8a5428.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 21:00:48,867 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e6b0db802b49804f36d630fc7a235f8b columnFamilyName info 2023-07-20 21:00:48,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7894): checking encryption for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:48,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(7897): checking classloading for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:48,868 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:48,870 DEBUG [StoreOpener-2587dac91101456d664c734eee8a5428-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f 2023-07-20 21:00:48,870 DEBUG [StoreOpener-2587dac91101456d664c734eee8a5428-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f 2023-07-20 21:00:48,870 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2587dac91101456d664c734eee8a5428 columnFamilyName f 2023-07-20 21:00:48,874 DEBUG [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/info/7105be2420954d3fb1f40ae1d0a97f75 2023-07-20 21:00:48,874 INFO [StoreOpener-e6b0db802b49804f36d630fc7a235f8b-1] regionserver.HStore(310): Store=e6b0db802b49804f36d630fc7a235f8b/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:48,875 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:48,879 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:48,880 DEBUG [StoreOpener-2587dac91101456d664c734eee8a5428-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/f/863335799b7b4a6ba9d31e57a9b6e34d 2023-07-20 21:00:48,880 INFO [StoreOpener-2587dac91101456d664c734eee8a5428-1] regionserver.HStore(310): Store=2587dac91101456d664c734eee8a5428/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 21:00:48,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428 2023-07-20 21:00:48,882 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:48,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428 2023-07-20 21:00:48,885 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-20 21:00:48,887 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened e6b0db802b49804f36d630fc7a235f8b; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64988645, jitterRate=-0.031593725085258484}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 21:00:48,887 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for e6b0db802b49804f36d630fc7a235f8b: 2023-07-20 21:00:48,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1055): writing seq id for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:48,888 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b., pid=33, masterSystemTime=1689886848858 2023-07-20 21:00:48,890 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:48,890 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:48,891 DEBUG [hconnection-0x71f5e0bf-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase16.apache.org/188.40.62.62:32921 this server is in the failed servers list 2023-07-20 21:00:48,891 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=e6b0db802b49804f36d630fc7a235f8b, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:48,892 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689886848891"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689886848891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689886848891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689886848891"}]},"ts":"1689886848891"} 2023-07-20 21:00:48,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-20 21:00:48,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1072): Opened 2587dac91101456d664c734eee8a5428; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71044634, jitterRate=0.05864754319190979}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 21:00:48,894 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(965): Region open journal for 2587dac91101456d664c734eee8a5428: 2023-07-20 21:00:48,894 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689886827417.2587dac91101456d664c734eee8a5428., pid=32, masterSystemTime=1689886848858 2023-07-20 21:00:48,896 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-07-20 21:00:48,897 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure e6b0db802b49804f36d630fc7a235f8b, server=jenkins-hbase16.apache.org,36285,1689886830049 in 189 msec 2023-07-20 21:00:48,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:48,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase16:0-0] handler.AssignRegionHandler(158): Opened t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:48,897 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=2587dac91101456d664c734eee8a5428, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:48,897 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689886827417.2587dac91101456d664c734eee8a5428.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689886848897"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689886848897"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689886848897"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689886848897"}]},"ts":"1689886848897"} 2023-07-20 21:00:48,898 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e6b0db802b49804f36d630fc7a235f8b, ASSIGN in 304 msec 2023-07-20 21:00:48,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=30 2023-07-20 21:00:48,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=30, state=SUCCESS; OpenRegionProcedure 2587dac91101456d664c734eee8a5428, server=jenkins-hbase16.apache.org,36285,1689886830049 in 195 msec 2023-07-20 21:00:48,902 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-07-20 21:00:48,902 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase16.apache.org,32921,1689886824455 after splitting done 2023-07-20 21:00:48,902 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=t, region=2587dac91101456d664c734eee8a5428, ASSIGN in 309 msec 2023-07-20 21:00:48,902 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase16.apache.org,32921,1689886824455 from processing; numProcessing=0 2023-07-20 21:00:48,904 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, state=SUCCESS; ServerCrashProcedure jenkins-hbase16.apache.org,32921,1689886824455, splitWal=true, meta=true in 1.2170 sec 2023-07-20 21:00:49,210 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dd2374a to 127.0.0.1:57031 2023-07-20 21:00:49,210 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:49,211 INFO [Listener at localhost.localdomain/46531] client.TestScannerTimeout(124): END ************ test2772 2023-07-20 21:00:49,223 INFO [Listener at localhost.localdomain/46531] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=280 (was 298), OpenFileDescriptor=582 (was 574) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=110 (was 120), ProcessCount=170 (was 170), AvailableMemoryMB=8814 (was 8820) 2023-07-20 21:00:49,224 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-20 21:00:49,224 INFO [Listener at localhost.localdomain/46531] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-20 21:00:49,224 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62eb3340 to 127.0.0.1:57031 2023-07-20 21:00:49,224 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:49,224 DEBUG [Listener at localhost.localdomain/46531] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-20 21:00:49,224 DEBUG [Listener at localhost.localdomain/46531] util.JVMClusterUtil(257): Found active master hash=1466667254, stopped=false 2023-07-20 21:00:49,224 INFO [Listener at localhost.localdomain/46531] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:49,247 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-20 21:00:49,247 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-20 21:00:49,248 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:49,248 INFO [Listener at localhost.localdomain/46531] procedure2.ProcedureExecutor(629): Stopping 2023-07-20 21:00:49,248 DEBUG [Listener at localhost.localdomain/46531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58dfe532 to 127.0.0.1:57031 2023-07-20 21:00:49,248 DEBUG [Listener at localhost.localdomain/46531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:49,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 21:00:49,248 INFO [Listener at localhost.localdomain/46531] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase16.apache.org,36285,1689886830049' ***** 2023-07-20 21:00:49,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 21:00:49,249 INFO [Listener at localhost.localdomain/46531] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-20 21:00:49,249 INFO [RS:2;jenkins-hbase16:36285] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 21:00:49,249 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 21:00:49,249 INFO [RS:2;jenkins-hbase16:36285] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-20 21:00:49,249 INFO [RS:2;jenkins-hbase16:36285] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-20 21:00:49,250 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(3305): Received CLOSE for 2587dac91101456d664c734eee8a5428 2023-07-20 21:00:49,250 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(3305): Received CLOSE for e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:49,250 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1144): stopping server jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:49,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 2587dac91101456d664c734eee8a5428, disabling compactions & flushes 2023-07-20 21:00:49,251 DEBUG [RS:2;jenkins-hbase16:36285] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2aa7af8b to 127.0.0.1:57031 2023-07-20 21:00:49,251 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:49,251 DEBUG [RS:2;jenkins-hbase16:36285] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:49,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:49,251 INFO [RS:2;jenkins-hbase16:36285] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 21:00:49,252 INFO [RS:2;jenkins-hbase16:36285] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 21:00:49,252 INFO [RS:2;jenkins-hbase16:36285] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 21:00:49,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689886827417.2587dac91101456d664c734eee8a5428. after waiting 0 ms 2023-07-20 21:00:49,252 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 21:00:49,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:49,252 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 21:00:49,252 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1478): Online Regions={2587dac91101456d664c734eee8a5428=t,,1689886827417.2587dac91101456d664c734eee8a5428., 1588230740=hbase:meta,,1.1588230740, e6b0db802b49804f36d630fc7a235f8b=hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b.} 2023-07-20 21:00:49,253 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 21:00:49,253 DEBUG [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1504): Waiting on 1588230740, 2587dac91101456d664c734eee8a5428, e6b0db802b49804f36d630fc7a235f8b 2023-07-20 21:00:49,253 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 21:00:49,253 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 21:00:49,255 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 21:00:49,255 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 21:00:49,256 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-20 21:00:49,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/default/t/2587dac91101456d664c734eee8a5428/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-20 21:00:49,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:49,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 2587dac91101456d664c734eee8a5428: 2023-07-20 21:00:49,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed t,,1689886827417.2587dac91101456d664c734eee8a5428. 2023-07-20 21:00:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1604): Closing e6b0db802b49804f36d630fc7a235f8b, disabling compactions & flushes 2023-07-20 21:00:49,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. after waiting 0 ms 2023-07-20 21:00:49,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:49,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/namespace/e6b0db802b49804f36d630fc7a235f8b/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-20 21:00:49,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:49,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for e6b0db802b49804f36d630fc7a235f8b: 2023-07-20 21:00:49,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689886826552.e6b0db802b49804f36d630fc7a235f8b. 2023-07-20 21:00:49,285 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=26 (bloomFilter=false), to=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/.tmp/info/2da95c58110849ed9c71e1953cd9d675 2023-07-20 21:00:49,294 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/.tmp/info/2da95c58110849ed9c71e1953cd9d675 as hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info/2da95c58110849ed9c71e1953cd9d675 2023-07-20 21:00:49,302 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/info/2da95c58110849ed9c71e1953cd9d675, entries=16, sequenceid=26, filesize=6.4 K 2023-07-20 21:00:49,303 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1796, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 48ms, sequenceid=26, compaction requested=true 2023-07-20 21:00:49,312 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/data/hbase/meta/1588230740/recovered.edits/29.seqid, newMaxSeqId=29, maxSeqId=19 2023-07-20 21:00:49,312 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 21:00:49,313 INFO [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 21:00:49,313 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 21:00:49,313 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase16:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 21:00:49,454 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1170): stopping server jenkins-hbase16.apache.org,36285,1689886830049; all regions closed. 2023-07-20 21:00:49,467 DEBUG [RS:2;jenkins-hbase16:36285] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs 2023-07-20 21:00:49,467 INFO [RS:2;jenkins-hbase16:36285] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C36285%2C1689886830049.meta:.meta(num 1689886848265) 2023-07-20 21:00:49,472 DEBUG [RS:2;jenkins-hbase16:36285] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/oldWALs 2023-07-20 21:00:49,473 INFO [RS:2;jenkins-hbase16:36285] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase16.apache.org%2C36285%2C1689886830049:(num 1689886830290) 2023-07-20 21:00:49,473 DEBUG [RS:2;jenkins-hbase16:36285] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:49,473 INFO [RS:2;jenkins-hbase16:36285] regionserver.LeaseManager(133): Closed leases 2023-07-20 21:00:49,473 INFO [RS:2;jenkins-hbase16:36285] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase16:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-20 21:00:49,473 INFO [regionserver/jenkins-hbase16:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 21:00:49,473 INFO [RS:2;jenkins-hbase16:36285] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:36285 2023-07-20 21:00:49,480 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:49,480 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase16.apache.org,36285,1689886830049 2023-07-20 21:00:49,480 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 21:00:49,488 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase16.apache.org,36285,1689886830049] 2023-07-20 21:00:49,488 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase16.apache.org,36285,1689886830049; numProcessing=1 2023-07-20 21:00:49,497 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase16.apache.org,36285,1689886830049 already deleted, retry=false 2023-07-20 21:00:49,497 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase16.apache.org,36285,1689886830049 expired; onlineServers=0 2023-07-20 21:00:49,497 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase16.apache.org,38535,1689886823355' ***** 2023-07-20 21:00:49,497 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-20 21:00:49,498 DEBUG [M:0;jenkins-hbase16:38535] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16569acd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase16.apache.org/188.40.62.62:0 2023-07-20 21:00:49,498 INFO [M:0;jenkins-hbase16:38535] regionserver.HRegionServer(1144): stopping server jenkins-hbase16.apache.org,38535,1689886823355 2023-07-20 21:00:49,498 INFO [M:0;jenkins-hbase16:38535] regionserver.HRegionServer(1170): stopping server jenkins-hbase16.apache.org,38535,1689886823355; all regions closed. 2023-07-20 21:00:49,498 DEBUG [M:0;jenkins-hbase16:38535] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 21:00:49,499 DEBUG [M:0;jenkins-hbase16:38535] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-20 21:00:49,499 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-20 21:00:49,499 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1689886825705] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.large.0-1689886825705,5,FailOnTimeoutGroup] 2023-07-20 21:00:49,499 DEBUG [master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1689886825707] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase16:0:becomeActiveMaster-HFileCleaner.small.0-1689886825707,5,FailOnTimeoutGroup] 2023-07-20 21:00:49,499 DEBUG [M:0;jenkins-hbase16:38535] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-20 21:00:49,500 INFO [M:0;jenkins-hbase16:38535] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-20 21:00:49,500 INFO [M:0;jenkins-hbase16:38535] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-20 21:00:49,500 INFO [M:0;jenkins-hbase16:38535] hbase.ChoreService(369): Chore service for: master/jenkins-hbase16:0 had [] on shutdown 2023-07-20 21:00:49,500 DEBUG [M:0;jenkins-hbase16:38535] master.HMaster(1512): Stopping service threads 2023-07-20 21:00:49,500 INFO [M:0;jenkins-hbase16:38535] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-20 21:00:49,501 ERROR [M:0;jenkins-hbase16:38535] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1870653254) connection to localhost.localdomain/127.0.0.1:36541 from jenkins,5,PEWorkerGroup] 2023-07-20 21:00:49,502 INFO [M:0;jenkins-hbase16:38535] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-20 21:00:49,502 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-20 21:00:49,510 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-20 21:00:49,510 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 21:00:49,510 DEBUG [M:0;jenkins-hbase16:38535] zookeeper.ZKUtil(398): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-20 21:00:49,510 WARN [M:0;jenkins-hbase16:38535] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-20 21:00:49,510 INFO [M:0;jenkins-hbase16:38535] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-20 21:00:49,510 INFO [M:0;jenkins-hbase16:38535] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-20 21:00:49,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 21:00:49,511 DEBUG [M:0;jenkins-hbase16:38535] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-20 21:00:49,512 INFO [M:0;jenkins-hbase16:38535] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 21:00:49,512 DEBUG [M:0;jenkins-hbase16:38535] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 21:00:49,512 DEBUG [M:0;jenkins-hbase16:38535] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-20 21:00:49,512 DEBUG [M:0;jenkins-hbase16:38535] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 21:00:49,512 INFO [M:0;jenkins-hbase16:38535] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=112.42 KB heapSize=135.38 KB 2023-07-20 21:00:49,537 INFO [M:0;jenkins-hbase16:38535] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=112.42 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/653fe28891774aacb0063da32296d16a 2023-07-20 21:00:49,545 DEBUG [M:0;jenkins-hbase16:38535] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/653fe28891774aacb0063da32296d16a as hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/653fe28891774aacb0063da32296d16a 2023-07-20 21:00:49,552 INFO [M:0;jenkins-hbase16:38535] regionserver.HStore(1080): Added hdfs://localhost.localdomain:36541/user/jenkins/test-data/3c8c5da6-bfeb-89da-6e0d-ff658a07e3ce/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/653fe28891774aacb0063da32296d16a, entries=33, sequenceid=257, filesize=8.5 K 2023-07-20 21:00:49,553 INFO [M:0;jenkins-hbase16:38535] regionserver.HRegion(2948): Finished flush of dataSize ~112.42 KB/115117, heapSize ~135.37 KB/138616, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 41ms, sequenceid=257, compaction requested=false 2023-07-20 21:00:49,554 INFO [M:0;jenkins-hbase16:38535] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 21:00:49,554 DEBUG [M:0;jenkins-hbase16:38535] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 21:00:49,558 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 21:00:49,558 INFO [M:0;jenkins-hbase16:38535] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-20 21:00:49,558 INFO [M:0;jenkins-hbase16:38535] ipc.NettyRpcServer(158): Stopping server on /188.40.62.62:38535 2023-07-20 21:00:49,568 DEBUG [M:0;jenkins-hbase16:38535] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase16.apache.org,38535,1689886823355 already deleted, retry=false 2023-07-20 21:00:49,740 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 21:00:49,740 INFO [M:0;jenkins-hbase16:38535] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase16.apache.org,38535,1689886823355; zookeeper connection closed. 2023-07-20 21:00:49,740 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): master:38535-0x1018488ceeb0000, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 21:00:49,841 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 21:00:49,841 DEBUG [Listener at localhost.localdomain/46531-EventThread] zookeeper.ZKWatcher(600): regionserver:36285-0x1018488ceeb0008, quorum=127.0.0.1:57031, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 21:00:49,841 INFO [RS:2;jenkins-hbase16:36285] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase16.apache.org,36285,1689886830049; zookeeper connection closed. 2023-07-20 21:00:49,841 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2aac46fc] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2aac46fc 2023-07-20 21:00:49,841 INFO [Listener at localhost.localdomain/46531] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-20 21:00:49,842 WARN [Listener at localhost.localdomain/46531] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-20 21:00:49,845 INFO [Listener at localhost.localdomain/46531] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 21:00:49,952 WARN [BP-1827106336-188.40.62.62-1689886819716 heartbeating to localhost.localdomain/127.0.0.1:36541] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-20 21:00:49,952 WARN [BP-1827106336-188.40.62.62-1689886819716 heartbeating to localhost.localdomain/127.0.0.1:36541] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1827106336-188.40.62.62-1689886819716 (Datanode Uuid bd4412a0-8085-425f-b4c6-c07ddf89b733) service to localhost.localdomain/127.0.0.1:36541 2023-07-20 21:00:49,954 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/dfs/data/data3/current/BP-1827106336-188.40.62.62-1689886819716] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 21:00:49,955 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/dfs/data/data4/current/BP-1827106336-188.40.62.62-1689886819716] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 21:00:49,957 WARN [Listener at localhost.localdomain/46531] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-20 21:00:49,959 INFO [Listener at localhost.localdomain/46531] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 21:00:50,070 WARN [BP-1827106336-188.40.62.62-1689886819716 heartbeating to localhost.localdomain/127.0.0.1:36541] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-20 21:00:50,070 WARN [BP-1827106336-188.40.62.62-1689886819716 heartbeating to localhost.localdomain/127.0.0.1:36541] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1827106336-188.40.62.62-1689886819716 (Datanode Uuid d04461f5-d76d-414b-9379-b0c97d8b86c5) service to localhost.localdomain/127.0.0.1:36541 2023-07-20 21:00:50,071 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/dfs/data/data1/current/BP-1827106336-188.40.62.62-1689886819716] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 21:00:50,071 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6d75c9b8-6523-b20a-54f1-a1d62be50b70/cluster_5c97f391-998a-3224-2d0a-75e791416fcf/dfs/data/data2/current/BP-1827106336-188.40.62.62-1689886819716] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 21:00:50,107 INFO [Listener at localhost.localdomain/46531] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-07-20 21:00:50,160 INFO [regionserver/jenkins-hbase16:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 21:00:50,225 INFO [Listener at localhost.localdomain/46531] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-20 21:00:50,258 INFO [Listener at localhost.localdomain/46531] hbase.HBaseTestingUtility(1293): Minicluster is down