2023-07-13 23:06:10,335 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-13 23:06:10,586 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552 2023-07-13 23:06:10,601 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-13 23:06:10,601 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6, deleteOnExit=true 2023-07-13 23:06:10,601 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-13 23:06:10,602 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/test.cache.data in system properties and HBase conf 2023-07-13 23:06:10,602 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/hadoop.tmp.dir in system properties and HBase conf 2023-07-13 23:06:10,603 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/hadoop.log.dir in system properties and HBase conf 2023-07-13 23:06:10,603 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-13 23:06:10,604 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-13 23:06:10,604 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-13 23:06:10,710 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-13 23:06:11,128 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-13 23:06:11,132 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-13 23:06:11,133 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-13 23:06:11,133 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-13 23:06:11,133 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-13 23:06:11,134 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-13 23:06:11,134 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-13 23:06:11,134 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-13 23:06:11,135 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-13 23:06:11,135 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-13 23:06:11,136 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/nfs.dump.dir in system properties and HBase conf 2023-07-13 23:06:11,136 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/java.io.tmpdir in system properties and HBase conf 2023-07-13 23:06:11,136 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-13 23:06:11,137 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-13 23:06:11,137 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-13 23:06:11,765 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-13 23:06:11,769 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-13 23:06:12,038 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-13 23:06:12,190 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-13 23:06:12,205 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-13 23:06:12,241 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-13 23:06:12,279 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/java.io.tmpdir/Jetty_localhost_42409_hdfs____g2sa7d/webapp 2023-07-13 23:06:12,410 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42409 2023-07-13 23:06:12,427 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-13 23:06:12,427 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-13 23:06:12,919 WARN [Listener at localhost/44535] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-13 23:06:12,986 WARN [Listener at localhost/44535] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-13 23:06:13,004 WARN [Listener at localhost/44535] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-13 23:06:13,011 INFO [Listener at localhost/44535] log.Slf4jLog(67): jetty-6.1.26 2023-07-13 23:06:13,016 INFO [Listener at localhost/44535] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/java.io.tmpdir/Jetty_localhost_46721_datanode____.qmq71h/webapp 2023-07-13 23:06:13,112 INFO [Listener at localhost/44535] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46721 2023-07-13 23:06:13,425 WARN [Listener at localhost/45803] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-13 23:06:13,437 WARN [Listener at localhost/45803] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-13 23:06:13,440 WARN [Listener at localhost/45803] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-13 23:06:13,442 INFO [Listener at localhost/45803] log.Slf4jLog(67): jetty-6.1.26 2023-07-13 23:06:13,447 INFO [Listener at localhost/45803] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/java.io.tmpdir/Jetty_localhost_44175_datanode____.7gjcqy/webapp 2023-07-13 23:06:13,544 INFO [Listener at localhost/45803] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44175 2023-07-13 23:06:13,554 WARN [Listener at localhost/39491] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-13 23:06:13,863 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd660cb0d36cef8ff: Processing first storage report for DS-4396672b-9e66-4761-b5f2-06f09255a1b9 from datanode 30ecc44a-0630-4f97-8054-35640690dc99 2023-07-13 23:06:13,864 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd660cb0d36cef8ff: from storage DS-4396672b-9e66-4761-b5f2-06f09255a1b9 node DatanodeRegistration(127.0.0.1:42655, datanodeUuid=30ecc44a-0630-4f97-8054-35640690dc99, infoPort=45681, infoSecurePort=0, ipcPort=39491, storageInfo=lv=-57;cid=testClusterID;nsid=1313507944;c=1689289571841), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-13 23:06:13,864 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd7369c9d95872ac1: Processing first storage report for DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15 from datanode ac436e4e-ea11-416f-b359-97c9d42b4361 2023-07-13 23:06:13,864 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd7369c9d95872ac1: from storage DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15 node DatanodeRegistration(127.0.0.1:35445, datanodeUuid=ac436e4e-ea11-416f-b359-97c9d42b4361, infoPort=33161, infoSecurePort=0, ipcPort=45803, storageInfo=lv=-57;cid=testClusterID;nsid=1313507944;c=1689289571841), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-13 23:06:13,865 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd660cb0d36cef8ff: Processing first storage report for DS-5aff9124-7bb2-4a0e-b86c-1a5ed09b51ce from datanode 30ecc44a-0630-4f97-8054-35640690dc99 2023-07-13 23:06:13,865 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd660cb0d36cef8ff: from storage DS-5aff9124-7bb2-4a0e-b86c-1a5ed09b51ce node DatanodeRegistration(127.0.0.1:42655, datanodeUuid=30ecc44a-0630-4f97-8054-35640690dc99, infoPort=45681, infoSecurePort=0, ipcPort=39491, storageInfo=lv=-57;cid=testClusterID;nsid=1313507944;c=1689289571841), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-13 23:06:13,865 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd7369c9d95872ac1: Processing first storage report for DS-f5f9d604-8d17-4319-844b-06fd2256bedf from datanode ac436e4e-ea11-416f-b359-97c9d42b4361 2023-07-13 23:06:13,865 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd7369c9d95872ac1: from storage DS-f5f9d604-8d17-4319-844b-06fd2256bedf node DatanodeRegistration(127.0.0.1:35445, datanodeUuid=ac436e4e-ea11-416f-b359-97c9d42b4361, infoPort=33161, infoSecurePort=0, ipcPort=45803, storageInfo=lv=-57;cid=testClusterID;nsid=1313507944;c=1689289571841), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-13 23:06:13,934 DEBUG [Listener at localhost/39491] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552 2023-07-13 23:06:13,996 INFO [Listener at localhost/39491] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/zookeeper_0, clientPort=57722, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-13 23:06:14,012 INFO [Listener at localhost/39491] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57722 2023-07-13 23:06:14,022 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:14,024 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:14,690 INFO [Listener at localhost/39491] util.FSUtils(471): Created version file at hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3 with version=8 2023-07-13 23:06:14,691 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/hbase-staging 2023-07-13 23:06:15,019 INFO [Listener at localhost/39491] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-13 23:06:15,502 INFO [Listener at localhost/39491] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-13 23:06:15,534 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:15,535 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:15,535 INFO [Listener at localhost/39491] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-13 23:06:15,535 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:15,535 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-13 23:06:15,675 INFO [Listener at localhost/39491] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-13 23:06:15,752 DEBUG [Listener at localhost/39491] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-13 23:06:15,844 INFO [Listener at localhost/39491] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34895 2023-07-13 23:06:15,854 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:15,856 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:15,877 INFO [Listener at localhost/39491] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34895 connecting to ZooKeeper ensemble=127.0.0.1:57722 2023-07-13 23:06:15,919 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:348950x0, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-13 23:06:15,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:34895-0x10160efc9610000 connected 2023-07-13 23:06:15,946 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-13 23:06:15,947 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 23:06:15,950 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-13 23:06:15,958 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34895 2023-07-13 23:06:15,958 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34895 2023-07-13 23:06:15,958 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34895 2023-07-13 23:06:15,959 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34895 2023-07-13 23:06:15,959 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34895 2023-07-13 23:06:15,965 INFO [Listener at localhost/39491] master.HMaster(444): hbase.rootdir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3, hbase.cluster.distributed=false 2023-07-13 23:06:16,030 INFO [Listener at localhost/39491] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-13 23:06:16,031 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:16,031 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:16,031 INFO [Listener at localhost/39491] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-13 23:06:16,031 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:16,031 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-13 23:06:16,036 INFO [Listener at localhost/39491] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-13 23:06:16,039 INFO [Listener at localhost/39491] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35415 2023-07-13 23:06:16,041 INFO [Listener at localhost/39491] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-13 23:06:16,047 DEBUG [Listener at localhost/39491] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-13 23:06:16,048 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:16,050 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:16,051 INFO [Listener at localhost/39491] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35415 connecting to ZooKeeper ensemble=127.0.0.1:57722 2023-07-13 23:06:16,055 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:354150x0, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-13 23:06:16,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:35415-0x10160efc9610001 connected 2023-07-13 23:06:16,056 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-13 23:06:16,058 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 23:06:16,058 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-13 23:06:16,059 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35415 2023-07-13 23:06:16,059 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35415 2023-07-13 23:06:16,060 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35415 2023-07-13 23:06:16,060 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35415 2023-07-13 23:06:16,061 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35415 2023-07-13 23:06:16,073 INFO [Listener at localhost/39491] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-13 23:06:16,073 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:16,073 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:16,074 INFO [Listener at localhost/39491] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-13 23:06:16,074 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:16,074 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-13 23:06:16,074 INFO [Listener at localhost/39491] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-13 23:06:16,076 INFO [Listener at localhost/39491] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46431 2023-07-13 23:06:16,076 INFO [Listener at localhost/39491] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-13 23:06:16,077 DEBUG [Listener at localhost/39491] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-13 23:06:16,078 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:16,080 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:16,082 INFO [Listener at localhost/39491] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46431 connecting to ZooKeeper ensemble=127.0.0.1:57722 2023-07-13 23:06:16,086 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:464310x0, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-13 23:06:16,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:46431-0x10160efc9610002 connected 2023-07-13 23:06:16,087 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-13 23:06:16,088 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 23:06:16,088 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-13 23:06:16,089 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46431 2023-07-13 23:06:16,089 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46431 2023-07-13 23:06:16,089 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46431 2023-07-13 23:06:16,090 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46431 2023-07-13 23:06:16,091 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46431 2023-07-13 23:06:16,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:16,100 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-13 23:06:16,101 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:16,121 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-13 23:06:16,121 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-13 23:06:16,121 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:16,121 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-13 23:06:16,122 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-13 23:06:16,124 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,34895,1689289574854 from backup master directory 2023-07-13 23:06:16,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-13 23:06:16,127 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:16,127 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-13 23:06:16,128 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-13 23:06:16,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:16,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-13 23:06:16,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-13 23:06:16,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/hbase.id with ID: 81f5c08a-03d5-44d3-ad49-97b00ec0d821 2023-07-13 23:06:16,260 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:16,274 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:16,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x515ca995 to 127.0.0.1:57722 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 23:06:16,346 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b2735c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 23:06:16,369 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-13 23:06:16,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-13 23:06:16,387 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-13 23:06:16,387 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-13 23:06:16,389 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-13 23:06:16,393 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-13 23:06:16,394 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 23:06:16,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store-tmp 2023-07-13 23:06:16,466 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:16,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-13 23:06:16,467 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 23:06:16,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 23:06:16,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-13 23:06:16,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 23:06:16,467 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 23:06:16,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-13 23:06:16,469 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/WALs/jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:16,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34895%2C1689289574854, suffix=, logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/WALs/jenkins-hbase4.apache.org,34895,1689289574854, archiveDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/oldWALs, maxLogs=10 2023-07-13 23:06:16,541 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK] 2023-07-13 23:06:16,541 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK] 2023-07-13 23:06:16,550 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 23:06:16,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/WALs/jenkins-hbase4.apache.org,34895,1689289574854/jenkins-hbase4.apache.org%2C34895%2C1689289574854.1689289576498 2023-07-13 23:06:16,612 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK], DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK]] 2023-07-13 23:06:16,613 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:16,613 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:16,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-13 23:06:16,617 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-13 23:06:16,672 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-13 23:06:16,678 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-13 23:06:16,702 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-13 23:06:16,713 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:16,719 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-13 23:06:16,720 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-13 23:06:16,734 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-13 23:06:16,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-13 23:06:16,738 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75090250, jitterRate=0.11893191933631897}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 23:06:16,738 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-13 23:06:16,740 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-13 23:06:16,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-13 23:06:16,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-13 23:06:16,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-13 23:06:16,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-13 23:06:16,794 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-13 23:06:16,794 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-13 23:06:16,818 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-13 23:06:16,824 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-13 23:06:16,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-13 23:06:16,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-13 23:06:16,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-13 23:06:16,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-13 23:06:16,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-13 23:06:16,867 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:16,868 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-13 23:06:16,869 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-13 23:06:16,879 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-13 23:06:16,884 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-13 23:06:16,884 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-13 23:06:16,884 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-13 23:06:16,884 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:16,885 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,34895,1689289574854, sessionid=0x10160efc9610000, setting cluster-up flag (Was=false) 2023-07-13 23:06:16,900 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:16,906 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-13 23:06:16,907 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:16,914 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:16,918 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-13 23:06:16,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:16,922 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.hbase-snapshot/.tmp 2023-07-13 23:06:16,994 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(951): ClusterId : 81f5c08a-03d5-44d3-ad49-97b00ec0d821 2023-07-13 23:06:16,994 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(951): ClusterId : 81f5c08a-03d5-44d3-ad49-97b00ec0d821 2023-07-13 23:06:16,999 DEBUG [RS:1;jenkins-hbase4:46431] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-13 23:06:16,999 DEBUG [RS:0;jenkins-hbase4:35415] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-13 23:06:17,007 DEBUG [RS:1;jenkins-hbase4:46431] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-13 23:06:17,007 DEBUG [RS:0;jenkins-hbase4:35415] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-13 23:06:17,007 DEBUG [RS:1;jenkins-hbase4:46431] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-13 23:06:17,007 DEBUG [RS:0;jenkins-hbase4:35415] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-13 23:06:17,011 DEBUG [RS:0;jenkins-hbase4:35415] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-13 23:06:17,011 DEBUG [RS:1;jenkins-hbase4:46431] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-13 23:06:17,012 DEBUG [RS:0;jenkins-hbase4:35415] zookeeper.ReadOnlyZKClient(139): Connect 0x40792ce3 to 127.0.0.1:57722 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 23:06:17,012 DEBUG [RS:1;jenkins-hbase4:46431] zookeeper.ReadOnlyZKClient(139): Connect 0x29a2d220 to 127.0.0.1:57722 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 23:06:17,028 DEBUG [RS:0;jenkins-hbase4:35415] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ebeda9c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 23:06:17,029 DEBUG [RS:1;jenkins-hbase4:46431] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27653964, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 23:06:17,029 DEBUG [RS:0;jenkins-hbase4:35415] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a814793, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-13 23:06:17,029 DEBUG [RS:1;jenkins-hbase4:46431] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@235ab950, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-13 23:06:17,036 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-13 23:06:17,047 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-13 23:06:17,047 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-13 23:06:17,047 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-13 23:06:17,047 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-13 23:06:17,047 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-13 23:06:17,047 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,048 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-13 23:06:17,048 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,051 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689289607051 2023-07-13 23:06:17,052 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35415 2023-07-13 23:06:17,053 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:46431 2023-07-13 23:06:17,053 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-13 23:06:17,056 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-13 23:06:17,056 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-13 23:06:17,057 INFO [RS:1;jenkins-hbase4:46431] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-13 23:06:17,057 INFO [RS:1;jenkins-hbase4:46431] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-13 23:06:17,057 INFO [RS:0;jenkins-hbase4:35415] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-13 23:06:17,057 INFO [RS:0;jenkins-hbase4:35415] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-13 23:06:17,057 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1022): About to register with Master. 2023-07-13 23:06:17,058 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1022): About to register with Master. 2023-07-13 23:06:17,060 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34895,1689289574854 with isa=jenkins-hbase4.apache.org/172.31.14.131:46431, startcode=1689289576073 2023-07-13 23:06:17,060 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34895,1689289574854 with isa=jenkins-hbase4.apache.org/172.31.14.131:35415, startcode=1689289576030 2023-07-13 23:06:17,061 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-13 23:06:17,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-13 23:06:17,073 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-13 23:06:17,073 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-13 23:06:17,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-13 23:06:17,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-13 23:06:17,074 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-13 23:06:17,079 DEBUG [RS:1;jenkins-hbase4:46431] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-13 23:06:17,079 DEBUG [RS:0;jenkins-hbase4:35415] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-13 23:06:17,080 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-13 23:06:17,080 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-13 23:06:17,083 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-13 23:06:17,084 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-13 23:06:17,092 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689289577086,5,FailOnTimeoutGroup] 2023-07-13 23:06:17,092 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689289577092,5,FailOnTimeoutGroup] 2023-07-13 23:06:17,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-13 23:06:17,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,116 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-13 23:06:17,117 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-13 23:06:17,117 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3 2023-07-13 23:06:17,141 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:17,145 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-13 23:06:17,148 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info 2023-07-13 23:06:17,149 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-13 23:06:17,150 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:17,151 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-13 23:06:17,151 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58055, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-13 23:06:17,151 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50815, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-13 23:06:17,153 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/rep_barrier 2023-07-13 23:06:17,154 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-13 23:06:17,155 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:17,155 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-13 23:06:17,157 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table 2023-07-13 23:06:17,158 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-13 23:06:17,159 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:17,160 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740 2023-07-13 23:06:17,161 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740 2023-07-13 23:06:17,163 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34895] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:17,165 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-13 23:06:17,168 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-13 23:06:17,172 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-13 23:06:17,173 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71418621, jitterRate=0.0642203837633133}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-13 23:06:17,173 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-13 23:06:17,173 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-13 23:06:17,173 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-13 23:06:17,173 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-13 23:06:17,173 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-13 23:06:17,173 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-13 23:06:17,174 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-13 23:06:17,174 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-13 23:06:17,175 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34895] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:17,180 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-13 23:06:17,180 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-13 23:06:17,182 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3 2023-07-13 23:06:17,182 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3 2023-07-13 23:06:17,182 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:44535 2023-07-13 23:06:17,182 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:44535 2023-07-13 23:06:17,182 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-13 23:06:17,182 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-13 23:06:17,187 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:17,188 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-13 23:06:17,192 DEBUG [RS:1;jenkins-hbase4:46431] zookeeper.ZKUtil(162): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:17,192 WARN [RS:1;jenkins-hbase4:46431] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-13 23:06:17,192 INFO [RS:1;jenkins-hbase4:46431] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 23:06:17,192 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1948): logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:17,193 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35415,1689289576030] 2023-07-13 23:06:17,193 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46431,1689289576073] 2023-07-13 23:06:17,193 DEBUG [RS:0;jenkins-hbase4:35415] zookeeper.ZKUtil(162): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:17,193 WARN [RS:0;jenkins-hbase4:35415] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-13 23:06:17,193 INFO [RS:0;jenkins-hbase4:35415] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 23:06:17,193 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1948): logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:17,203 DEBUG [RS:1;jenkins-hbase4:46431] zookeeper.ZKUtil(162): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:17,203 DEBUG [RS:0;jenkins-hbase4:35415] zookeeper.ZKUtil(162): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:17,204 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-13 23:06:17,204 DEBUG [RS:1;jenkins-hbase4:46431] zookeeper.ZKUtil(162): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:17,206 DEBUG [RS:0;jenkins-hbase4:35415] zookeeper.ZKUtil(162): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:17,207 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-13 23:06:17,215 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-13 23:06:17,215 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-13 23:06:17,224 INFO [RS:1;jenkins-hbase4:46431] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-13 23:06:17,224 INFO [RS:0;jenkins-hbase4:35415] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-13 23:06:17,244 INFO [RS:0;jenkins-hbase4:35415] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-13 23:06:17,244 INFO [RS:1;jenkins-hbase4:46431] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-13 23:06:17,248 INFO [RS:0;jenkins-hbase4:35415] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-13 23:06:17,248 INFO [RS:1;jenkins-hbase4:46431] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-13 23:06:17,248 INFO [RS:0;jenkins-hbase4:35415] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,248 INFO [RS:1;jenkins-hbase4:46431] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,249 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-13 23:06:17,249 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-13 23:06:17,257 INFO [RS:1;jenkins-hbase4:46431] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,257 INFO [RS:0;jenkins-hbase4:35415] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,257 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,257 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-13 23:06:17,258 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-13 23:06:17,258 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,258 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,259 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,259 DEBUG [RS:1;jenkins-hbase4:46431] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,259 DEBUG [RS:0;jenkins-hbase4:35415] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:17,260 INFO [RS:1;jenkins-hbase4:46431] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,260 INFO [RS:0;jenkins-hbase4:35415] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,260 INFO [RS:1;jenkins-hbase4:46431] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,260 INFO [RS:0;jenkins-hbase4:35415] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,260 INFO [RS:1;jenkins-hbase4:46431] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,261 INFO [RS:0;jenkins-hbase4:35415] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,278 INFO [RS:1;jenkins-hbase4:46431] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-13 23:06:17,278 INFO [RS:0;jenkins-hbase4:35415] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-13 23:06:17,281 INFO [RS:0;jenkins-hbase4:35415] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35415,1689289576030-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,281 INFO [RS:1;jenkins-hbase4:46431] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46431,1689289576073-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,304 INFO [RS:0;jenkins-hbase4:35415] regionserver.Replication(203): jenkins-hbase4.apache.org,35415,1689289576030 started 2023-07-13 23:06:17,304 INFO [RS:1;jenkins-hbase4:46431] regionserver.Replication(203): jenkins-hbase4.apache.org,46431,1689289576073 started 2023-07-13 23:06:17,304 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,35415,1689289576030, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35415, sessionid=0x10160efc9610001 2023-07-13 23:06:17,304 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,46431,1689289576073, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46431, sessionid=0x10160efc9610002 2023-07-13 23:06:17,304 DEBUG [RS:0;jenkins-hbase4:35415] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-13 23:06:17,304 DEBUG [RS:1;jenkins-hbase4:46431] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-13 23:06:17,304 DEBUG [RS:0;jenkins-hbase4:35415] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:17,304 DEBUG [RS:1;jenkins-hbase4:46431] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:17,305 DEBUG [RS:0;jenkins-hbase4:35415] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35415,1689289576030' 2023-07-13 23:06:17,305 DEBUG [RS:1;jenkins-hbase4:46431] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46431,1689289576073' 2023-07-13 23:06:17,305 DEBUG [RS:1;jenkins-hbase4:46431] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-13 23:06:17,305 DEBUG [RS:0;jenkins-hbase4:35415] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-13 23:06:17,307 DEBUG [RS:1;jenkins-hbase4:46431] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-13 23:06:17,307 DEBUG [RS:0;jenkins-hbase4:35415] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-13 23:06:17,308 DEBUG [RS:1;jenkins-hbase4:46431] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-13 23:06:17,308 DEBUG [RS:0;jenkins-hbase4:35415] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-13 23:06:17,308 DEBUG [RS:1;jenkins-hbase4:46431] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-13 23:06:17,308 DEBUG [RS:0;jenkins-hbase4:35415] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-13 23:06:17,308 DEBUG [RS:1;jenkins-hbase4:46431] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:17,308 DEBUG [RS:0;jenkins-hbase4:35415] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:17,308 DEBUG [RS:0;jenkins-hbase4:35415] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35415,1689289576030' 2023-07-13 23:06:17,308 DEBUG [RS:0;jenkins-hbase4:35415] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-13 23:06:17,308 DEBUG [RS:1;jenkins-hbase4:46431] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46431,1689289576073' 2023-07-13 23:06:17,308 DEBUG [RS:1;jenkins-hbase4:46431] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-13 23:06:17,309 DEBUG [RS:1;jenkins-hbase4:46431] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-13 23:06:17,309 DEBUG [RS:0;jenkins-hbase4:35415] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-13 23:06:17,309 DEBUG [RS:1;jenkins-hbase4:46431] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-13 23:06:17,310 INFO [RS:1;jenkins-hbase4:46431] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-13 23:06:17,310 DEBUG [RS:0;jenkins-hbase4:35415] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-13 23:06:17,310 INFO [RS:1;jenkins-hbase4:46431] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-13 23:06:17,310 INFO [RS:0;jenkins-hbase4:35415] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-13 23:06:17,310 INFO [RS:0;jenkins-hbase4:35415] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-13 23:06:17,359 DEBUG [jenkins-hbase4:34895] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-13 23:06:17,364 DEBUG [jenkins-hbase4:34895] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-13 23:06:17,370 DEBUG [jenkins-hbase4:34895] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-13 23:06:17,370 DEBUG [jenkins-hbase4:34895] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-13 23:06:17,370 DEBUG [jenkins-hbase4:34895] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-13 23:06:17,373 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35415,1689289576030, state=OPENING 2023-07-13 23:06:17,380 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-13 23:06:17,381 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:17,382 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 23:06:17,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35415,1689289576030}] 2023-07-13 23:06:17,420 INFO [RS:0;jenkins-hbase4:35415] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35415%2C1689289576030, suffix=, logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030, archiveDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs, maxLogs=32 2023-07-13 23:06:17,420 INFO [RS:1;jenkins-hbase4:46431] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46431%2C1689289576073, suffix=, logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073, archiveDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs, maxLogs=32 2023-07-13 23:06:17,442 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK] 2023-07-13 23:06:17,444 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK] 2023-07-13 23:06:17,447 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK] 2023-07-13 23:06:17,447 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK] 2023-07-13 23:06:17,454 INFO [RS:1;jenkins-hbase4:46431] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425 2023-07-13 23:06:17,454 INFO [RS:0;jenkins-hbase4:35415] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425 2023-07-13 23:06:17,454 DEBUG [RS:1;jenkins-hbase4:46431] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK], DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK]] 2023-07-13 23:06:17,454 DEBUG [RS:0;jenkins-hbase4:35415] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK], DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK]] 2023-07-13 23:06:17,569 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:17,572 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-13 23:06:17,575 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-13 23:06:17,587 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-13 23:06:17,587 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 23:06:17,591 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta, suffix=.meta, logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030, archiveDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs, maxLogs=32 2023-07-13 23:06:17,608 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK] 2023-07-13 23:06:17,608 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK] 2023-07-13 23:06:17,614 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta 2023-07-13 23:06:17,621 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK], DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK]] 2023-07-13 23:06:17,621 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:17,623 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-13 23:06:17,638 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-13 23:06:17,670 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-13 23:06:17,675 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-13 23:06:17,675 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:17,675 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-13 23:06:17,676 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-13 23:06:17,680 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-13 23:06:17,682 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info 2023-07-13 23:06:17,682 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info 2023-07-13 23:06:17,682 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-13 23:06:17,683 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:17,683 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-13 23:06:17,685 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/rep_barrier 2023-07-13 23:06:17,685 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/rep_barrier 2023-07-13 23:06:17,685 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-13 23:06:17,686 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:17,686 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-13 23:06:17,688 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table 2023-07-13 23:06:17,688 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table 2023-07-13 23:06:17,688 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-13 23:06:17,689 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:17,691 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740 2023-07-13 23:06:17,693 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740 2023-07-13 23:06:17,696 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-13 23:06:17,698 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-13 23:06:17,700 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70609980, jitterRate=0.05217069387435913}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-13 23:06:17,700 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-13 23:06:17,709 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689289577562 2023-07-13 23:06:17,726 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-13 23:06:17,727 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-13 23:06:17,727 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35415,1689289576030, state=OPEN 2023-07-13 23:06:17,733 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-13 23:06:17,733 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 23:06:17,737 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-13 23:06:17,737 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35415,1689289576030 in 347 msec 2023-07-13 23:06:17,742 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-13 23:06:17,742 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 550 msec 2023-07-13 23:06:17,748 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 780 msec 2023-07-13 23:06:17,748 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689289577748, completionTime=-1 2023-07-13 23:06:17,748 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-13 23:06:17,748 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-13 23:06:17,823 DEBUG [hconnection-0x5e4b45fd-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:17,826 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:17,841 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-13 23:06:17,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689289637842 2023-07-13 23:06:17,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689289697842 2023-07-13 23:06:17,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 93 msec 2023-07-13 23:06:17,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34895,1689289574854-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34895,1689289574854-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34895,1689289574854-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:34895, period=300000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:17,871 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-13 23:06:17,879 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-13 23:06:17,881 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-13 23:06:17,891 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-13 23:06:17,893 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-13 23:06:17,895 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-13 23:06:17,916 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:17,918 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 empty. 2023-07-13 23:06:17,919 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:17,919 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-13 23:06:17,945 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-13 23:06:17,948 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 539033c7f1670a68ba89b4a328a12c36, NAME => 'hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp 2023-07-13 23:06:17,963 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:17,963 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 539033c7f1670a68ba89b4a328a12c36, disabling compactions & flushes 2023-07-13 23:06:17,963 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:17,963 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:17,963 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. after waiting 0 ms 2023-07-13 23:06:17,963 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:17,963 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:17,963 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 539033c7f1670a68ba89b4a328a12c36: 2023-07-13 23:06:17,967 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-13 23:06:17,982 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689289577970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689289577970"}]},"ts":"1689289577970"} 2023-07-13 23:06:18,011 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-13 23:06:18,013 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-13 23:06:18,018 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689289578014"}]},"ts":"1689289578014"} 2023-07-13 23:06:18,021 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-13 23:06:18,026 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-13 23:06:18,028 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-13 23:06:18,028 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-13 23:06:18,028 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-13 23:06:18,030 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN}] 2023-07-13 23:06:18,033 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN 2023-07-13 23:06:18,035 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35415,1689289576030; forceNewPlan=false, retain=false 2023-07-13 23:06:18,187 INFO [jenkins-hbase4:34895] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-13 23:06:18,188 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=539033c7f1670a68ba89b4a328a12c36, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:18,189 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689289578188"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689289578188"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689289578188"}]},"ts":"1689289578188"} 2023-07-13 23:06:18,194 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 539033c7f1670a68ba89b4a328a12c36, server=jenkins-hbase4.apache.org,35415,1689289576030}] 2023-07-13 23:06:18,354 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:18,355 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 539033c7f1670a68ba89b4a328a12c36, NAME => 'hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:18,356 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:18,356 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:18,356 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:18,356 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:18,358 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:18,360 DEBUG [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info 2023-07-13 23:06:18,360 DEBUG [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info 2023-07-13 23:06:18,361 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539033c7f1670a68ba89b4a328a12c36 columnFamilyName info 2023-07-13 23:06:18,362 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] regionserver.HStore(310): Store=539033c7f1670a68ba89b4a328a12c36/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:18,363 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:18,364 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:18,368 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:18,371 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-13 23:06:18,372 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 539033c7f1670a68ba89b4a328a12c36; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60916629, jitterRate=-0.09227149188518524}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 23:06:18,372 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 539033c7f1670a68ba89b4a328a12c36: 2023-07-13 23:06:18,374 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36., pid=6, masterSystemTime=1689289578348 2023-07-13 23:06:18,378 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:18,378 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:18,379 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=539033c7f1670a68ba89b4a328a12c36, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:18,379 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689289578379"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689289578379"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689289578379"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689289578379"}]},"ts":"1689289578379"} 2023-07-13 23:06:18,387 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-13 23:06:18,387 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 539033c7f1670a68ba89b4a328a12c36, server=jenkins-hbase4.apache.org,35415,1689289576030 in 189 msec 2023-07-13 23:06:18,391 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-13 23:06:18,391 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN in 357 msec 2023-07-13 23:06:18,392 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-13 23:06:18,392 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689289578392"}]},"ts":"1689289578392"} 2023-07-13 23:06:18,395 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-13 23:06:18,399 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-13 23:06:18,402 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 517 msec 2023-07-13 23:06:18,493 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-13 23:06:18,494 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-13 23:06:18,495 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:18,530 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-13 23:06:18,546 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-13 23:06:18,553 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-07-13 23:06:18,563 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-13 23:06:18,573 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-13 23:06:18,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-07-13 23:06:18,590 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-13 23:06:18,592 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-13 23:06:18,594 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.466sec 2023-07-13 23:06:18,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-13 23:06:18,598 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-13 23:06:18,598 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-13 23:06:18,599 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34895,1689289574854-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-13 23:06:18,599 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34895,1689289574854-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-13 23:06:18,601 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(139): Connect 0x2f246984 to 127.0.0.1:57722 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 23:06:18,605 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65b7854e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 23:06:18,610 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-13 23:06:18,619 DEBUG [hconnection-0x35b7c425-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:18,629 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:18,638 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:18,647 DEBUG [Listener at localhost/39491] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-13 23:06:18,649 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59958, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-13 23:06:18,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-13 23:06:18,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-13 23:06:18,665 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-13 23:06:18,667 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-13 23:06:18,669 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:18,670 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd empty. 2023-07-13 23:06:18,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-13 23:06:18,672 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:18,672 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-13 23:06:18,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 23:06:18,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 23:06:18,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 23:06:19,093 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-13 23:06:19,094 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5a1d70f57ebb5ff6152a7364d4d54cdd, NAME => 't,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/.tmp 2023-07-13 23:06:19,111 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:19,111 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 5a1d70f57ebb5ff6152a7364d4d54cdd, disabling compactions & flushes 2023-07-13 23:06:19,111 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,111 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,111 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. after waiting 0 ms 2023-07-13 23:06:19,111 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,111 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,111 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 5a1d70f57ebb5ff6152a7364d4d54cdd: 2023-07-13 23:06:19,115 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-13 23:06:19,117 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689289579116"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689289579116"}]},"ts":"1689289579116"} 2023-07-13 23:06:19,119 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-13 23:06:19,120 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-13 23:06:19,120 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689289579120"}]},"ts":"1689289579120"} 2023-07-13 23:06:19,122 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-13 23:06:19,127 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-13 23:06:19,128 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-13 23:06:19,128 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-13 23:06:19,128 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-13 23:06:19,128 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN}] 2023-07-13 23:06:19,130 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN 2023-07-13 23:06:19,131 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35415,1689289576030; forceNewPlan=false, retain=false 2023-07-13 23:06:19,281 INFO [jenkins-hbase4:34895] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-13 23:06:19,282 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=5a1d70f57ebb5ff6152a7364d4d54cdd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:19,283 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689289579282"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689289579282"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689289579282"}]},"ts":"1689289579282"} 2023-07-13 23:06:19,285 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 5a1d70f57ebb5ff6152a7364d4d54cdd, server=jenkins-hbase4.apache.org,35415,1689289576030}] 2023-07-13 23:06:19,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 23:06:19,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5a1d70f57ebb5ff6152a7364d4d54cdd, NAME => 't,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:19,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:19,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,448 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,450 DEBUG [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f 2023-07-13 23:06:19,450 DEBUG [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f 2023-07-13 23:06:19,450 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5a1d70f57ebb5ff6152a7364d4d54cdd columnFamilyName f 2023-07-13 23:06:19,451 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] regionserver.HStore(310): Store=5a1d70f57ebb5ff6152a7364d4d54cdd/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:19,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-13 23:06:19,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5a1d70f57ebb5ff6152a7364d4d54cdd; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61371719, jitterRate=-0.08549012243747711}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 23:06:19,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5a1d70f57ebb5ff6152a7364d4d54cdd: 2023-07-13 23:06:19,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd., pid=11, masterSystemTime=1689289579438 2023-07-13 23:06:19,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,467 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=5a1d70f57ebb5ff6152a7364d4d54cdd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:19,467 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689289579467"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689289579467"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689289579467"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689289579467"}]},"ts":"1689289579467"} 2023-07-13 23:06:19,473 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-13 23:06:19,473 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 5a1d70f57ebb5ff6152a7364d4d54cdd, server=jenkins-hbase4.apache.org,35415,1689289576030 in 185 msec 2023-07-13 23:06:19,476 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-13 23:06:19,476 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN in 345 msec 2023-07-13 23:06:19,477 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-13 23:06:19,477 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689289579477"}]},"ts":"1689289579477"} 2023-07-13 23:06:19,479 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-13 23:06:19,482 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-13 23:06:19,484 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 821 msec 2023-07-13 23:06:19,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-13 23:06:19,793 INFO [Listener at localhost/39491] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-13 23:06:19,793 DEBUG [Listener at localhost/39491] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-13 23:06:19,794 INFO [Listener at localhost/39491] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-13 23:06:19,798 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-13 23:06:19,799 INFO [Listener at localhost/39491] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-13 23:06:19,799 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-13 23:06:19,846 INFO [Listener at localhost/39491] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=293, OpenFileDescriptor=581, MaxFileDescriptor=60000, SystemLoadAverage=177, ProcessCount=170, AvailableMemoryMB=7089 2023-07-13 23:06:19,848 INFO [Listener at localhost/39491] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-13 23:06:19,851 DEBUG [Listener at localhost/39491] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-13 23:06:19,852 DEBUG [Listener at localhost/39491] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,852 INFO [Listener at localhost/39491] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-13 23:06:19,852 INFO [Listener at localhost/39491] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-13 23:06:19,856 INFO [Listener at localhost/39491] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689289578392/Put/vlen=2/seqid=0} 2023-07-13 23:06:19,856 INFO [Listener at localhost/39491] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-13 23:06:19,856 INFO [Listener at localhost/39491] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36./info:regioninfo/1689289578379/Put/vlen=41/seqid=0, hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36./info:seqnumDuringOpen/1689289578379/Put/vlen=8/seqid=0, hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36./info:server/1689289578379/Put/vlen=31/seqid=0, hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36./info:serverstartcode/1689289578379/Put/vlen=8/seqid=0, hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36./info:sn/1689289578188/Put/vlen=45/seqid=0, hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36./info:state/1689289578379/Put/vlen=4/seqid=0} 2023-07-13 23:06:19,856 INFO [Listener at localhost/39491] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 539033c7f1670a68ba89b4a328a12c36, NAME => 'hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:19,856 INFO [Listener at localhost/39491] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689289579477/Put/vlen=2/seqid=0} 2023-07-13 23:06:19,857 INFO [Listener at localhost/39491] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-13 23:06:19,857 INFO [Listener at localhost/39491] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd./info:regioninfo/1689289579467/Put/vlen=35/seqid=0, t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd./info:seqnumDuringOpen/1689289579467/Put/vlen=8/seqid=0, t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd./info:server/1689289579467/Put/vlen=31/seqid=0, t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd./info:serverstartcode/1689289579467/Put/vlen=8/seqid=0, t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd./info:sn/1689289579282/Put/vlen=45/seqid=0, t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd./info:state/1689289579467/Put/vlen=4/seqid=0} 2023-07-13 23:06:19,857 INFO [Listener at localhost/39491] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 5a1d70f57ebb5ff6152a7364d4d54cdd, NAME => 't,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:19,858 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(139): Connect 0x63f4beb1 to 127.0.0.1:57722 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 23:06:19,862 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@255f57ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 23:06:19,862 INFO [Listener at localhost/39491] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-13 23:06:19,862 INFO [Listener at localhost/39491] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-13 23:06:19,865 DEBUG [hconnection-0x28d47ff0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:19,868 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:19,873 INFO [Listener at localhost/39491] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-13 23:06:19,875 ERROR [Listener at localhost/39491] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,35415,1689289576030: die! ***** 2023-07-13 23:06:19,876 ERROR [Listener at localhost/39491] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-13 23:06:19,879 DEBUG [Listener at localhost/39491] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-13 23:06:19,881 DEBUG [Listener at localhost/39491] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-13 23:06:19,882 DEBUG [Listener at localhost/39491] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-13 23:06:19,882 DEBUG [Listener at localhost/39491] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-13 23:06:19,883 INFO [Listener at localhost/39491] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1123549184, "init": 513802240, "max": 2051014656, "used": 477590056 }, "NonHeapMemoryUsage": { "committed": 113270784, "init": 2555904, "max": -1, "used": 111242488 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-13 23:06:19,886 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34895] master.MasterRpcServices(612): jenkins-hbase4.apache.org,35415,1689289576030 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,35415,1689289576030: die! ***** 2023-07-13 23:06:19,888 INFO [Listener at localhost/39491] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,35415,1689289576030' ***** 2023-07-13 23:06:19,888 INFO [Listener at localhost/39491] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-13 23:06:19,889 INFO [RS:0;jenkins-hbase4:35415] regionserver.HeapMemoryManager(220): Stopping 2023-07-13 23:06:19,889 INFO [RS:0;jenkins-hbase4:35415] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-13 23:06:19,889 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-13 23:06:19,890 INFO [RS:0;jenkins-hbase4:35415] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-13 23:06:19,890 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(3305): Received CLOSE for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:19,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35415] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 7400659268724064268 2023-07-13 23:06:19,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35415] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35415,1689289576030 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 23:06:19,890 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(3305): Received CLOSE for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35415] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:50228 deadline: 1689289639890, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35415,1689289576030 aborting 2023-07-13 23:06:19,891 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:19,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 539033c7f1670a68ba89b4a328a12c36, disabling compactions & flushes 2023-07-13 23:06:19,891 DEBUG [RS:0;jenkins-hbase4:35415] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40792ce3 to 127.0.0.1:57722 2023-07-13 23:06:19,891 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:19,891 DEBUG [RS:0;jenkins-hbase4:35415] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:19,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:19,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. after waiting 0 ms 2023-07-13 23:06:19,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:19,892 INFO [RS:0;jenkins-hbase4:35415] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-13 23:06:19,892 INFO [RS:0;jenkins-hbase4:35415] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-13 23:06:19,892 INFO [RS:0;jenkins-hbase4:35415] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-13 23:06:19,892 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-13 23:06:19,892 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:19,892 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-13 23:06:19,893 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:19,893 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-13 23:06:19,893 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1478): Online Regions={539033c7f1670a68ba89b4a328a12c36=hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36., 5a1d70f57ebb5ff6152a7364d4d54cdd=t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd., 1588230740=hbase:meta,,1.1588230740} 2023-07-13 23:06:19,893 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-13 23:06:19,893 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 539033c7f1670a68ba89b4a328a12c36: 2023-07-13 23:06:19,893 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-13 23:06:19,893 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-13 23:06:19,893 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-13 23:06:19,893 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:19,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5a1d70f57ebb5ff6152a7364d4d54cdd, disabling compactions & flushes 2023-07-13 23:06:19,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. after waiting 0 ms 2023-07-13 23:06:19,894 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-13 23:06:19,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,895 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-13 23:06:19,895 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,896 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-13 23:06:19,896 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-13 23:06:19,896 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-13 23:06:19,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5a1d70f57ebb5ff6152a7364d4d54cdd: 2023-07-13 23:06:19,896 DEBUG [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1504): Waiting on 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:19,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:19,910 DEBUG [Listener at localhost/39491] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35415,1689289576030 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35415,1689289576030 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-13 23:06:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35415] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35415,1689289576030 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 23:06:19,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35415] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:50228 deadline: 1689289639912, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35415,1689289576030 aborting 2023-07-13 23:06:20,015 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35415] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35415,1689289576030 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 23:06:20,016 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=35415] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:50228 deadline: 1689289640015, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,35415,1689289576030 aborting 2023-07-13 23:06:20,097 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,35415,1689289576030; all regions closed. 2023-07-13 23:06:20,111 DEBUG [RS:0;jenkins-hbase4:35415] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:20,112 INFO [RS:0;jenkins-hbase4:35415] regionserver.LeaseManager(133): Closed leases 2023-07-13 23:06:20,112 INFO [RS:0;jenkins-hbase4:35415] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-13 23:06:20,112 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-13 23:06:20,113 INFO [RS:0;jenkins-hbase4:35415] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35415 2023-07-13 23:06:20,121 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35415 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35415 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 23:06:20,122 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35415 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35415 2023-07-13 23:06:20,122 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:20,122 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:20,122 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:20,122 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:20,122 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:20,124 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35415,1689289576030] 2023-07-13 23:06:20,124 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35415,1689289576030; numProcessing=1 2023-07-13 23:06:20,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:20,125 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,35415,1689289576030 znode expired, triggering replicatorRemoved event 2023-07-13 23:06:20,126 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35415,1689289576030 already deleted, retry=false 2023-07-13 23:06:20,126 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,35415,1689289576030 on jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:20,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:20,134 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,35415,1689289576030, splitWal=true, meta=true 2023-07-13 23:06:20,134 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,35415,1689289576030 (carryingMeta=true) jenkins-hbase4.apache.org,35415,1689289576030/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@f9a9805[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-13 23:06:20,136 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35415,1689289576030, splitWal=true, meta=true 2023-07-13 23:06:20,138 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35415,1689289576030, splitWal=true, meta=true, isMeta: true 2023-07-13 23:06:20,140 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting 2023-07-13 23:06:20,141 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,35415,1689289576030 WAL count=1, meta=true 2023-07-13 23:06:20,144 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta}] 2023-07-13 23:06:20,153 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:20,155 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta, worker=jenkins-hbase4.apache.org,46431,1689289576073}] 2023-07-13 23:06:20,224 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 23:06:20,225 INFO [RS:0;jenkins-hbase4:35415] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,35415,1689289576030; zookeeper connection closed. 2023-07-13 23:06:20,225 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:35415-0x10160efc9610001, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 23:06:20,225 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@af1b4d2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@af1b4d2 2023-07-13 23:06:20,263 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-13 23:06:20,318 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:20,318 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-13 23:06:20,321 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36550, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-13 23:06:20,322 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46431] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-13 23:06:20,324 DEBUG [hconnection-0x28d47ff0-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35415 this server is in the failed servers list 2023-07-13 23:06:20,344 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta, size=3.1 K (3185bytes) 2023-07-13 23:06:20,345 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta 2023-07-13 23:06:20,350 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta after 5ms 2023-07-13 23:06:20,355 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta took 12ms 2023-07-13 23:06:20,371 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-13 23:06:20,373 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta so closing down 2023-07-13 23:06:20,373 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-13 23:06:20,377 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-13 23:06:20,380 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta.temp 2023-07-13 23:06:20,381 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-13 23:06:20,391 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-13 23:06:20,393 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta.temp to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-13 23:06:20,393 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 37 ms; skipped=1; WAL=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-13 23:06:20,394 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta, journal: Splitting hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta, size=3.1 K (3185bytes) at 1689289580344Finishing writing output for hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta so closing down at 1689289580373 (+29 ms)Creating recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta.temp at 1689289580380 (+7 ms)3 split writer threads finished at 1689289580381 (+1 ms)Closed recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689289580391 (+10 ms)Rename recovered edits hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta.temp to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689289580393 (+2 ms)Processed 11 edits across 1 Regions in 37 ms; skipped=1; WAL=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689289580394 (+1 ms) 2023-07-13 23:06:20,394 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta 2023-07-13 23:06:20,396 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-13 23:06:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.HMaster(3641): Remote procedure done, pid=14 2023-07-13 23:06:20,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] wal.WALSplitUtil(141): Moved hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs 2023-07-13 23:06:20,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-13 23:06:20,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta, worker=jenkins-hbase4.apache.org,46431,1689289576073 in 252 msec 2023-07-13 23:06:20,411 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:20,415 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-13 23:06:20,415 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C35415%2C1689289576030.meta.1689289577592.meta, worker=jenkins-hbase4.apache.org,46431,1689289576073 in 269 msec 2023-07-13 23:06:20,418 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting dir is empty, no logs to split. 2023-07-13 23:06:20,418 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,35415,1689289576030 WAL count=0, meta=true 2023-07-13 23:06:20,418 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,35415,1689289576030 WAL splitting is done? wals=0, meta=true 2023-07-13 23:06:20,419 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-13 23:06:20,421 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-13 23:06:20,422 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 23:06:20,572 DEBUG [jenkins-hbase4:34895] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-13 23:06:20,572 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46431,1689289576073, state=OPENING 2023-07-13 23:06:20,575 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-13 23:06:20,575 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 23:06:20,575 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46431,1689289576073}] 2023-07-13 23:06:20,627 WARN [ReadOnlyZKClient-127.0.0.1:57722@0x63f4beb1] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-13 23:06:20,627 DEBUG [hconnection-0x28d47ff0-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:20,630 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36560, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:20,631 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46431] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:36560 deadline: 1689289640631, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:20,734 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-13 23:06:20,734 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 23:06:20,736 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta, suffix=.meta, logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073, archiveDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs, maxLogs=32 2023-07-13 23:06:20,757 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK] 2023-07-13 23:06:20,757 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK] 2023-07-13 23:06:20,761 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta 2023-07-13 23:06:20,761 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK], DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK]] 2023-07-13 23:06:20,761 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:20,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-13 23:06:20,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-13 23:06:20,762 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-13 23:06:20,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-13 23:06:20,762 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:20,763 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-13 23:06:20,763 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-13 23:06:20,765 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-13 23:06:20,766 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info 2023-07-13 23:06:20,766 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info 2023-07-13 23:06:20,767 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-13 23:06:20,767 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:20,768 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-13 23:06:20,769 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/rep_barrier 2023-07-13 23:06:20,769 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/rep_barrier 2023-07-13 23:06:20,770 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-13 23:06:20,771 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:20,771 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-13 23:06:20,772 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table 2023-07-13 23:06:20,772 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table 2023-07-13 23:06:20,772 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-13 23:06:20,773 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:20,775 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740 2023-07-13 23:06:20,778 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740 2023-07-13 23:06:20,780 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-13 23:06:20,792 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-13 23:06:20,792 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-13 23:06:20,847 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/.tmp/info/16f9c5e413d74c93a3d7c19c644a051f 2023-07-13 23:06:20,903 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/.tmp/table/958f287388b54f1aa4ce9daa93b9a06b 2023-07-13 23:06:20,912 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/.tmp/info/16f9c5e413d74c93a3d7c19c644a051f as hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info/16f9c5e413d74c93a3d7c19c644a051f 2023-07-13 23:06:20,921 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info/16f9c5e413d74c93a3d7c19c644a051f, entries=20, sequenceid=13, filesize=6.9 K 2023-07-13 23:06:20,923 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/.tmp/table/958f287388b54f1aa4ce9daa93b9a06b as hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table/958f287388b54f1aa4ce9daa93b9a06b 2023-07-13 23:06:20,931 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table/958f287388b54f1aa4ce9daa93b9a06b, entries=4, sequenceid=13, filesize=4.7 K 2023-07-13 23:06:20,931 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 139ms, sequenceid=13, compaction requested=false; wal=null 2023-07-13 23:06:20,932 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-13 23:06:20,936 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-13 23:06:20,938 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-13 23:06:20,940 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-13 23:06:20,943 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-13 23:06:20,944 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65915694, jitterRate=-0.017779618501663208}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-13 23:06:20,945 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-13 23:06:20,946 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689289580728 2023-07-13 23:06:20,948 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-13 23:06:20,949 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-13 23:06:20,949 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46431,1689289576073, state=OPEN 2023-07-13 23:06:20,951 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-13 23:06:20,951 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 23:06:20,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-13 23:06:20,954 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46431,1689289576073 in 376 msec 2023-07-13 23:06:20,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-13 23:06:20,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 535 msec 2023-07-13 23:06:20,956 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,35415,1689289576030 had 3 regions 2023-07-13 23:06:20,958 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,35415,1689289576030, splitWal=true, meta=true, isMeta: false 2023-07-13 23:06:20,960 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,35415,1689289576030 WAL count=1, meta=false 2023-07-13 23:06:20,960 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425}] 2023-07-13 23:06:20,962 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:20,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425, worker=jenkins-hbase4.apache.org,46431,1689289576073}] 2023-07-13 23:06:21,116 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46431] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-13 23:06:21,132 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425, size=1.6 K (1665bytes) 2023-07-13 23:06:21,132 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425 2023-07-13 23:06:21,133 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425 after 1ms 2023-07-13 23:06:21,137 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425 took 6ms 2023-07-13 23:06:21,141 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 539033c7f1670a68ba89b4a328a12c36: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-13 23:06:21,144 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 5a1d70f57ebb5ff6152a7364d4d54cdd: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-13 23:06:21,145 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425 so closing down 2023-07-13 23:06:21,145 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-13 23:06:21,147 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp 2023-07-13 23:06:21,149 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp 2023-07-13 23:06:21,149 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-13 23:06:21,172 DEBUG [hconnection-0x28d47ff0-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35415 this server is in the failed servers list 2023-07-13 23:06:21,174 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-13 23:06:21,176 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000005 2023-07-13 23:06:21,379 DEBUG [hconnection-0x28d47ff0-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35415 this server is in the failed servers list 2023-07-13 23:06:21,575 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-13 23:06:21,577 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000013 2023-07-13 23:06:21,577 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 440 ms; skipped=2; WAL=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-13 23:06:21,578 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425, journal: Splitting hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425, size=1.6 K (1665bytes) at 1689289581132Finishing writing output for hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425 so closing down at 1689289581145 (+13 ms)Creating recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp at 1689289581147 (+2 ms)Creating recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp at 1689289581149 (+2 ms)3 split writer threads finished at 1689289581150 (+1 ms)Closed recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689289581174 (+24 ms)Rename recovered edits hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000005 at 1689289581176 (+2 ms)Closed recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689289581575 (+399 ms)Rename recovered edits hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425.temp to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000013 at 1689289581577 (+2 ms)Processed 14 edits across 2 Regions in 440 ms; skipped=2; WAL=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689289581578 (+1 ms) 2023-07-13 23:06:21,578 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425 2023-07-13 23:06:21,578 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-13 23:06:21,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.HMaster(3641): Remote procedure done, pid=18 2023-07-13 23:06:21,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] wal.WALSplitUtil(141): Moved hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting/jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425 to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs 2023-07-13 23:06:21,587 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-13 23:06:21,587 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425, worker=jenkins-hbase4.apache.org,46431,1689289576073 in 621 msec 2023-07-13 23:06:21,589 DEBUG [PEWorker-4] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:21,591 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-13 23:06:21,591 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C35415%2C1689289576030.1689289577425, worker=jenkins-hbase4.apache.org,46431,1689289576073 in 629 msec 2023-07-13 23:06:21,593 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,35415,1689289576030-splitting dir is empty, no logs to split. 2023-07-13 23:06:21,593 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,35415,1689289576030 WAL count=0, meta=false 2023-07-13 23:06:21,593 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,35415,1689289576030 WAL splitting is done? wals=0, meta=false 2023-07-13 23:06:21,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN}, {pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN}] 2023-07-13 23:06:21,597 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN 2023-07-13 23:06:21,597 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN 2023-07-13 23:06:21,598 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 23:06:21,598 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 23:06:21,687 DEBUG [hconnection-0x28d47ff0-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:35415 this server is in the failed servers list 2023-07-13 23:06:21,748 DEBUG [jenkins-hbase4:34895] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-13 23:06:21,749 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=5a1d70f57ebb5ff6152a7364d4d54cdd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:21,749 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=539033c7f1670a68ba89b4a328a12c36, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:21,749 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689289581749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689289581749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689289581749"}]},"ts":"1689289581749"} 2023-07-13 23:06:21,749 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689289581749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689289581749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689289581749"}]},"ts":"1689289581749"} 2023-07-13 23:06:21,751 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35415 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35415 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 23:06:21,752 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35415 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35415 2023-07-13 23:06:21,855 DEBUG [PEWorker-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:21,857 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:21,859 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=19, state=RUNNABLE; OpenRegionProcedure 539033c7f1670a68ba89b4a328a12c36, server=jenkins-hbase4.apache.org,46431,1689289576073}] 2023-07-13 23:06:21,860 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=20, state=RUNNABLE; OpenRegionProcedure 5a1d70f57ebb5ff6152a7364d4d54cdd, server=jenkins-hbase4.apache.org,46431,1689289576073}] 2023-07-13 23:06:22,017 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:22,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5a1d70f57ebb5ff6152a7364d4d54cdd, NAME => 't,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:22,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:22,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:22,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:22,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:22,020 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:22,021 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:22,022 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 539033c7f1670a68ba89b4a328a12c36, NAME => 'hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:22,022 DEBUG [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f 2023-07-13 23:06:22,022 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:22,022 DEBUG [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f 2023-07-13 23:06:22,022 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:22,022 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:22,023 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:22,023 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5a1d70f57ebb5ff6152a7364d4d54cdd columnFamilyName f 2023-07-13 23:06:22,024 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] regionserver.HStore(310): Store=5a1d70f57ebb5ff6152a7364d4d54cdd/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:22,025 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:22,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:22,027 DEBUG [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info 2023-07-13 23:06:22,027 DEBUG [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info 2023-07-13 23:06:22,028 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539033c7f1670a68ba89b4a328a12c36 columnFamilyName info 2023-07-13 23:06:22,029 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] regionserver.HStore(310): Store=539033c7f1670a68ba89b4a328a12c36/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:22,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:22,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000013 2023-07-13 23:06:22,034 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:22,038 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:22,039 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000005 2023-07-13 23:06:22,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000013 2023-07-13 23:06:22,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5a1d70f57ebb5ff6152a7364d4d54cdd 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-13 23:06:22,046 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000005 2023-07-13 23:06:22,047 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 539033c7f1670a68ba89b4a328a12c36 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-13 23:06:22,062 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/.tmp/f/d43b3cf4ae4640c18f3ec2e20d787916 2023-07-13 23:06:22,072 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/.tmp/info/35734b3eca764bacae8802aff91a418e 2023-07-13 23:06:22,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/.tmp/f/d43b3cf4ae4640c18f3ec2e20d787916 as hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f/d43b3cf4ae4640c18f3ec2e20d787916 2023-07-13 23:06:22,087 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/.tmp/info/35734b3eca764bacae8802aff91a418e as hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info/35734b3eca764bacae8802aff91a418e 2023-07-13 23:06:22,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f/d43b3cf4ae4640c18f3ec2e20d787916, entries=10, sequenceid=13, filesize=4.8 K 2023-07-13 23:06:22,089 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 5a1d70f57ebb5ff6152a7364d4d54cdd in 49ms, sequenceid=13, compaction requested=false; wal=null 2023-07-13 23:06:22,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-13 23:06:22,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/0000000000000000013 2023-07-13 23:06:22,096 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info/35734b3eca764bacae8802aff91a418e, entries=2, sequenceid=5, filesize=4.8 K 2023-07-13 23:06:22,097 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 539033c7f1670a68ba89b4a328a12c36 in 50ms, sequenceid=5, compaction requested=false; wal=null 2023-07-13 23:06:22,097 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-13 23:06:22,111 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/0000000000000000005 2023-07-13 23:06:22,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:22,116 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:22,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-13 23:06:22,123 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-13 23:06:22,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5a1d70f57ebb5ff6152a7364d4d54cdd; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65317369, jitterRate=-0.026695355772972107}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 23:06:22,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5a1d70f57ebb5ff6152a7364d4d54cdd: 2023-07-13 23:06:22,125 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 539033c7f1670a68ba89b4a328a12c36; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65870422, jitterRate=-0.01845422387123108}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 23:06:22,125 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 539033c7f1670a68ba89b4a328a12c36: 2023-07-13 23:06:22,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd., pid=22, masterSystemTime=1689289582011 2023-07-13 23:06:22,127 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36., pid=21, masterSystemTime=1689289582011 2023-07-13 23:06:22,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:22,129 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:22,130 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=5a1d70f57ebb5ff6152a7364d4d54cdd, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:22,130 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:22,130 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689289582130"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689289582130"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689289582130"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689289582130"}]},"ts":"1689289582130"} 2023-07-13 23:06:22,131 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:22,131 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=539033c7f1670a68ba89b4a328a12c36, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:22,132 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689289582131"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689289582131"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689289582131"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689289582131"}]},"ts":"1689289582131"} 2023-07-13 23:06:22,136 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=20 2023-07-13 23:06:22,136 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=20, state=SUCCESS; OpenRegionProcedure 5a1d70f57ebb5ff6152a7364d4d54cdd, server=jenkins-hbase4.apache.org,46431,1689289576073 in 273 msec 2023-07-13 23:06:22,137 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=19 2023-07-13 23:06:22,138 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=19, state=SUCCESS; OpenRegionProcedure 539033c7f1670a68ba89b4a328a12c36, server=jenkins-hbase4.apache.org,46431,1689289576073 in 275 msec 2023-07-13 23:06:22,139 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN in 540 msec 2023-07-13 23:06:22,141 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-07-13 23:06:22,141 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN in 542 msec 2023-07-13 23:06:22,141 INFO [PEWorker-2] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,35415,1689289576030 after splitting done 2023-07-13 23:06:22,141 DEBUG [PEWorker-2] master.DeadServer(114): Removed jenkins-hbase4.apache.org,35415,1689289576030 from processing; numProcessing=0 2023-07-13 23:06:22,145 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,35415,1689289576030, splitWal=true, meta=true in 2.0140 sec 2023-07-13 23:06:22,202 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63f4beb1 to 127.0.0.1:57722 2023-07-13 23:06:22,202 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:22,202 INFO [Listener at localhost/39491] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-13 23:06:22,215 INFO [Listener at localhost/39491] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=268 (was 293), OpenFileDescriptor=583 (was 581) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=177 (was 177), ProcessCount=170 (was 170), AvailableMemoryMB=7032 (was 7089) 2023-07-13 23:06:22,226 INFO [Listener at localhost/39491] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=268, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=177, ProcessCount=170, AvailableMemoryMB=7032 2023-07-13 23:06:22,227 INFO [Listener at localhost/39491] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:35415 2023-07-13 23:06:22,239 INFO [Listener at localhost/39491] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-13 23:06:22,239 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:22,239 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:22,239 INFO [Listener at localhost/39491] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-13 23:06:22,239 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-13 23:06:22,239 INFO [Listener at localhost/39491] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-13 23:06:22,239 INFO [Listener at localhost/39491] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-13 23:06:22,241 INFO [Listener at localhost/39491] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46589 2023-07-13 23:06:22,241 INFO [Listener at localhost/39491] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-13 23:06:22,242 DEBUG [Listener at localhost/39491] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-13 23:06:22,242 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:22,244 INFO [Listener at localhost/39491] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-13 23:06:22,245 INFO [Listener at localhost/39491] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46589 connecting to ZooKeeper ensemble=127.0.0.1:57722 2023-07-13 23:06:22,248 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:465890x0, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-13 23:06:22,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:46589-0x10160efc9610008 connected 2023-07-13 23:06:22,250 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(162): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-13 23:06:22,250 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(162): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-13 23:06:22,251 DEBUG [Listener at localhost/39491] zookeeper.ZKUtil(164): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-13 23:06:22,253 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46589 2023-07-13 23:06:22,253 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46589 2023-07-13 23:06:22,253 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46589 2023-07-13 23:06:22,253 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46589 2023-07-13 23:06:22,254 DEBUG [Listener at localhost/39491] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46589 2023-07-13 23:06:22,255 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(951): ClusterId : 81f5c08a-03d5-44d3-ad49-97b00ec0d821 2023-07-13 23:06:22,256 DEBUG [RS:2;jenkins-hbase4:46589] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-13 23:06:22,259 DEBUG [RS:2;jenkins-hbase4:46589] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-13 23:06:22,259 DEBUG [RS:2;jenkins-hbase4:46589] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-13 23:06:22,261 DEBUG [RS:2;jenkins-hbase4:46589] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-13 23:06:22,262 DEBUG [RS:2;jenkins-hbase4:46589] zookeeper.ReadOnlyZKClient(139): Connect 0x0c4a9c37 to 127.0.0.1:57722 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 23:06:22,266 DEBUG [RS:2;jenkins-hbase4:46589] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28d63ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 23:06:22,266 DEBUG [RS:2;jenkins-hbase4:46589] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b2f747, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-13 23:06:22,276 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:46589 2023-07-13 23:06:22,276 INFO [RS:2;jenkins-hbase4:46589] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-13 23:06:22,276 INFO [RS:2;jenkins-hbase4:46589] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-13 23:06:22,276 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1022): About to register with Master. 2023-07-13 23:06:22,277 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34895,1689289574854 with isa=jenkins-hbase4.apache.org/172.31.14.131:46589, startcode=1689289582238 2023-07-13 23:06:22,277 DEBUG [RS:2;jenkins-hbase4:46589] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-13 23:06:22,279 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33389, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-13 23:06:22,280 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34895] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:22,280 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3 2023-07-13 23:06:22,280 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:44535 2023-07-13 23:06:22,281 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-13 23:06:22,282 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:22,282 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:22,283 DEBUG [RS:2;jenkins-hbase4:46589] zookeeper.ZKUtil(162): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:22,283 WARN [RS:2;jenkins-hbase4:46589] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-13 23:06:22,283 INFO [RS:2;jenkins-hbase4:46589] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 23:06:22,283 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46589,1689289582238] 2023-07-13 23:06:22,283 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1948): logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:22,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:22,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:22,287 DEBUG [RS:2;jenkins-hbase4:46589] zookeeper.ZKUtil(162): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:22,288 DEBUG [RS:2;jenkins-hbase4:46589] zookeeper.ZKUtil(162): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:22,289 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-13 23:06:22,289 INFO [RS:2;jenkins-hbase4:46589] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-13 23:06:22,292 INFO [RS:2;jenkins-hbase4:46589] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-13 23:06:22,293 INFO [RS:2;jenkins-hbase4:46589] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-13 23:06:22,293 INFO [RS:2;jenkins-hbase4:46589] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:22,293 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-13 23:06:22,295 INFO [RS:2;jenkins-hbase4:46589] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:22,295 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,296 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,296 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,296 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,296 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,296 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-13 23:06:22,296 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,296 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,296 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,297 DEBUG [RS:2;jenkins-hbase4:46589] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-13 23:06:22,299 INFO [RS:2;jenkins-hbase4:46589] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:22,299 INFO [RS:2;jenkins-hbase4:46589] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:22,299 INFO [RS:2;jenkins-hbase4:46589] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:22,311 INFO [RS:2;jenkins-hbase4:46589] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-13 23:06:22,311 INFO [RS:2;jenkins-hbase4:46589] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46589,1689289582238-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-13 23:06:22,323 INFO [RS:2;jenkins-hbase4:46589] regionserver.Replication(203): jenkins-hbase4.apache.org,46589,1689289582238 started 2023-07-13 23:06:22,323 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,46589,1689289582238, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46589, sessionid=0x10160efc9610008 2023-07-13 23:06:22,323 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:46589,5,FailOnTimeoutGroup] 2023-07-13 23:06:22,323 DEBUG [RS:2;jenkins-hbase4:46589] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-13 23:06:22,323 DEBUG [RS:2;jenkins-hbase4:46589] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:22,323 DEBUG [RS:2;jenkins-hbase4:46589] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46589,1689289582238' 2023-07-13 23:06:22,323 DEBUG [RS:2;jenkins-hbase4:46589] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-13 23:06:22,323 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:22,323 INFO [Listener at localhost/39491] client.TestScannerTimeout(177): START ************ test3686b 2023-07-13 23:06:22,324 DEBUG [RS:2;jenkins-hbase4:46589] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-13 23:06:22,324 DEBUG [RS:2;jenkins-hbase4:46589] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-13 23:06:22,324 DEBUG [RS:2;jenkins-hbase4:46589] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-13 23:06:22,324 DEBUG [RS:2;jenkins-hbase4:46589] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:22,324 DEBUG [RS:2;jenkins-hbase4:46589] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46589,1689289582238' 2023-07-13 23:06:22,324 DEBUG [RS:2;jenkins-hbase4:46589] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-13 23:06:22,325 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:35415 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35415 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 23:06:22,325 DEBUG [RS:2;jenkins-hbase4:46589] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-13 23:06:22,326 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:35415 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:35415 2023-07-13 23:06:22,327 DEBUG [RS:2;jenkins-hbase4:46589] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-13 23:06:22,327 INFO [RS:2;jenkins-hbase4:46589] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-13 23:06:22,327 INFO [RS:2;jenkins-hbase4:46589] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-13 23:06:22,428 DEBUG [hconnection-0x35b7c425-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:22,430 INFO [RS:2;jenkins-hbase4:46589] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46589%2C1689289582238, suffix=, logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46589,1689289582238, archiveDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs, maxLogs=32 2023-07-13 23:06:22,431 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36580, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:22,434 DEBUG [Listener at localhost/39491] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-13 23:06:22,434 DEBUG [Listener at localhost/39491] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:22,435 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(139): Connect 0x1a2ecda0 to 127.0.0.1:57722 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 23:06:22,440 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41b30ab8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 23:06:22,443 DEBUG [hconnection-0x76039b2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:22,446 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:22,458 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK] 2023-07-13 23:06:22,458 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK] 2023-07-13 23:06:22,467 INFO [RS:2;jenkins-hbase4:46589] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46589,1689289582238/jenkins-hbase4.apache.org%2C46589%2C1689289582238.1689289582432 2023-07-13 23:06:22,468 DEBUG [RS:2;jenkins-hbase4:46589] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK], DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK]] 2023-07-13 23:06:23,186 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-13 23:06:25,015 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-13 23:06:25,016 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-13 23:06:37,454 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 735803557948686344 expired clientIPAndPort=172.31.14.131:36586, userName=jenkins, regionInfo=t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:39,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46431] regionserver.RSRpcServices(3094): Client tried to access missing scanner 735803557948686344 2023-07-13 23:06:39,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46431] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:36586 deadline: 1689289659455, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '735803557948686344'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-13 23:06:39,457 DEBUG [Listener at localhost/39491] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '735803557948686344'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '735803557948686344'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '735803557948686344'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-13 23:06:39,462 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a2ecda0 to 127.0.0.1:57722 2023-07-13 23:06:39,462 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:39,462 INFO [Listener at localhost/39491] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-13 23:06:39,482 INFO [Listener at localhost/39491] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=297 (was 268) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57722@0x1a2ecda0-SendThread(127.0.0.1:57722) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:44535 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-515644697-172.31.14.131-1689289571841:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-515644697-172.31.14.131-1689289571841:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-117902031_17 at /127.0.0.1:55264 [Receiving block BP-515644697-172.31.14.131-1689289571841:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:46589 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-117902031_17 at /127.0.0.1:35064 [Receiving block BP-515644697-172.31.14.131-1689289571841:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57722@0x1a2ecda0-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57722@0x0c4a9c37-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57722@0x0c4a9c37-SendThread(127.0.0.1:57722) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57722@0x1a2ecda0 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1815088080.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: jenkins-hbase4:46589Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:46589-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57722@0x0c4a9c37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1815088080.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=46589 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3-prefix:jenkins-hbase4.apache.org,46589,1689289582238 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=574 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=139 (was 177), ProcessCount=170 (was 170), AvailableMemoryMB=6925 (was 7032) 2023-07-13 23:06:39,497 INFO [Listener at localhost/39491] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=297, OpenFileDescriptor=574, MaxFileDescriptor=60000, SystemLoadAverage=139, ProcessCount=170, AvailableMemoryMB=6924 2023-07-13 23:06:39,497 INFO [Listener at localhost/39491] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:35415 2023-07-13 23:06:39,497 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,35415,1689289576030 2023-07-13 23:06:39,497 INFO [Listener at localhost/39491] client.TestScannerTimeout(104): START************ test2772 2023-07-13 23:06:39,501 DEBUG [Listener at localhost/39491] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-13 23:06:39,501 DEBUG [Listener at localhost/39491] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:39,501 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(139): Connect 0x1103e632 to 127.0.0.1:57722 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-13 23:06:39,506 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71842cc4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-13 23:06:39,506 ERROR [Listener at localhost/39491] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,46431,1689289576073: die! ***** 2023-07-13 23:06:39,506 ERROR [Listener at localhost/39491] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-13 23:06:39,506 DEBUG [Listener at localhost/39491] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-13 23:06:39,507 DEBUG [Listener at localhost/39491] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-13 23:06:39,512 DEBUG [Listener at localhost/39491] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-13 23:06:39,513 DEBUG [Listener at localhost/39491] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-13 23:06:39,528 INFO [Listener at localhost/39491] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1124073472, "init": 513802240, "max": 2051014656, "used": 206962232 }, "NonHeapMemoryUsage": { "committed": 124149760, "init": 2555904, "max": -1, "used": 121637832 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13457, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 65, "ProcessCallTime_min": 1, "ProcessCallTime_max": 2, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 2, "ProcessCallTime_75th_percentile": 2, "ProcessCallTime_90th_percentile": 2, "ProcessCallTime_95th_percentile": 2, "ProcessCallTime_98th_percentile": 2, "ProcessCallTime_99th_percentile": 2, "ProcessCallTime_99.9th_percentile": 2, "ProcessCallTime_TimeRangeCount_0-1": 1, "ProcessCallTime_TimeRangeCount_1-3": 1, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 65, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 65, "TotalCallTime_min": 1, "TotalCallTime_max": 2, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 2, "TotalCallTime_75th_percentile": 2, "TotalCallTime_90th_percentile": 2, "TotalCallTime_95th_percentile": 2, "TotalCallTime_98th_percentile": 2, "TotalCallTime_99th_percentile": 2, "TotalCallTime_99.9th_percentile": 2, "TotalCallTime_TimeRangeCount_0-1": 1, "TotalCallTime_TimeRangeCount_1-3": 1, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 65, "ResponseSize_min": 196, "ResponseSize_max": 678, "ResponseSize_mean": 437, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 196, "ResponseSize_75th_percentile": 675, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 677, "ResponseSize_98th_percentile": 677, "ResponseSize_99th_percentile": 677, "ResponseSize_99.9th_percentile": 677, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 5, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 1, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 65, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 17675 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:57722", "tag.serverName": "jenkins-hbase4.apache.org,46431,1689289576073", "tag.clusterId": "81f5c08a-03d5-44d3-ad49-97b00ec0d821", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 3, "storeCount": 5, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 4, "maxStoreFileCount": 1, "memStoreSize": 1792, "storeFileSize": 21711, "maxStoreFileAge": 16382, "minStoreFileAge": 15157, "avgStoreFileAge": 15758, "numReferenceFiles": 0, "regionServerStartTime": 1689289576073, "averageRegionSize": 7834, "storeFileIndexSize": 1368, "staticIndexSize": 221, "staticBloomSize": 4, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2492, "localBytesRead": 2492, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819799248, "blockCacheCount": 3, "blockCacheDataBlockCount": 3, "blockCacheSize": 606640, "blockCacheCountHitPercent": 87.5, "blockCacheExpressHitPercent": 87.5, "l1CacheSize": 606640, "l1CacheFreeSize": 819799248, "l1CacheCount": 3, "l1CacheEvictionCount": 0, "l1CacheHitCount": 21, "l1CacheMissCount": 3, "l1CacheHitRatio": 0.875, "l1CacheMissRatio": 0.125, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 8750, "ByteBuffAllocatorPoolAllocationBytes": 931840, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 20, "totalRowActionRequestCount": 24, "readRequestCount": 20, "filteredReadRequestCount": 0, "writeRequestCount": 4, "rpcGetRequestCount": 5, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 11, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 4, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 21, "blockCacheHitCountPrimary": 21, "blockCacheHitCachingCount": 21, "blockCacheMissCount": 3, "blockCacheMissCountPrimary": 3, "blockCacheMissCachingCount": 3, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 3, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 21, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 36, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 2742, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 10, "ScanTime_min": 0, "ScanTime_max": 1, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 1, "ScanTime_75th_percentile": 1, "ScanTime_90th_percentile": 1, "ScanTime_95th_percentile": 1, "ScanTime_98th_percentile": 1, "ScanTime_99th_percentile": 1, "ScanTime_99.9th_percentile": 1, "ScanTime_TimeRangeCount_0-1": 2, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 4, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 3, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 4, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 21711, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 3, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 10, "ScanSize_min": 155, "ScanSize_max": 653, "ScanSize_mean": 404, "ScanSize_25th_percentile": 155, "ScanSize_median": 155, "ScanSize_75th_percentile": 653, "ScanSize_90th_percentile": 653, "ScanSize_95th_percentile": 653, "ScanSize_98th_percentile": 653, "ScanSize_99th_percentile": 653, "ScanSize_99.9th_percentile": 653, "ScanSize_SizeRangeCount_100-1000": 2, "slowGetCount": 0, "flushedMemstoreBytes": 2742, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 3, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.423457754968975, "ServerReadQueryPerSecond_1min_rate": 6.480919739732413, "ServerReadQueryPerSecond_5min_rate": 7.822462627279663, "ServerReadQueryPerSecond_15min_rate": 8.07213643099849, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.166849973928216, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-13 23:06:39,529 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34895] master.MasterRpcServices(612): jenkins-hbase4.apache.org,46431,1689289576073 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,46431,1689289576073: die! ***** 2023-07-13 23:06:39,530 INFO [Listener at localhost/39491] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,46431,1689289576073' ***** 2023-07-13 23:06:39,530 INFO [Listener at localhost/39491] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-13 23:06:39,531 INFO [RS:1;jenkins-hbase4:46431] regionserver.HeapMemoryManager(220): Stopping 2023-07-13 23:06:39,531 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-13 23:06:39,531 INFO [RS:1;jenkins-hbase4:46431] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-13 23:06:39,531 INFO [RS:1;jenkins-hbase4:46431] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-13 23:06:39,531 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(3305): Received CLOSE for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:39,531 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(3305): Received CLOSE for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:39,532 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:39,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 539033c7f1670a68ba89b4a328a12c36, disabling compactions & flushes 2023-07-13 23:06:39,532 DEBUG [RS:1;jenkins-hbase4:46431] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29a2d220 to 127.0.0.1:57722 2023-07-13 23:06:39,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:39,532 DEBUG [RS:1;jenkins-hbase4:46431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:39,532 INFO [RS:1;jenkins-hbase4:46431] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-13 23:06:39,532 INFO [RS:1;jenkins-hbase4:46431] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-13 23:06:39,532 INFO [RS:1;jenkins-hbase4:46431] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-13 23:06:39,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:39,533 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-13 23:06:39,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. after waiting 0 ms 2023-07-13 23:06:39,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:39,538 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-13 23:06:39,538 DEBUG [hconnection-0x3de720da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:39,539 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-13 23:06:39,538 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1478): Online Regions={539033c7f1670a68ba89b4a328a12c36=hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36., 5a1d70f57ebb5ff6152a7364d4d54cdd=t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd., 1588230740=hbase:meta,,1.1588230740} 2023-07-13 23:06:39,539 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-13 23:06:39,539 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-13 23:06:39,539 DEBUG [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1504): Waiting on 1588230740, 539033c7f1670a68ba89b4a328a12c36, 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:39,539 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-13 23:06:39,539 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-13 23:06:39,546 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-13 23:06:39,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:39,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 539033c7f1670a68ba89b4a328a12c36: 2023-07-13 23:06:39,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:39,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5a1d70f57ebb5ff6152a7364d4d54cdd, disabling compactions & flushes 2023-07-13 23:06:39,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:39,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:39,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. after waiting 0 ms 2023-07-13 23:06:39,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:39,547 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:39,548 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46431] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46431,1689289576073 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 23:06:39,548 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46431] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:46122 deadline: 1689289659548, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46431,1689289576073 aborting 2023-07-13 23:06:39,550 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 1792 in region hbase:meta,,1.1588230740 2023-07-13 23:06:39,551 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-13 23:06:39,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:39,552 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-13 23:06:39,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5a1d70f57ebb5ff6152a7364d4d54cdd: 2023-07-13 23:06:39,552 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-13 23:06:39,552 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-13 23:06:39,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:39,655 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46431] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46431,1689289576073 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-13 23:06:39,655 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=46431] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:46122 deadline: 1689289659654, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,46431,1689289576073 aborting 2023-07-13 23:06:39,739 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,46431,1689289576073; all regions closed. 2023-07-13 23:06:39,763 DEBUG [RS:1;jenkins-hbase4:46431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:39,763 INFO [RS:1;jenkins-hbase4:46431] regionserver.LeaseManager(133): Closed leases 2023-07-13 23:06:39,763 INFO [RS:1;jenkins-hbase4:46431] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-13 23:06:39,764 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-13 23:06:39,765 INFO [RS:1;jenkins-hbase4:46431] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46431 2023-07-13 23:06:39,772 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:39,772 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:39,772 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:39,773 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 2023-07-13 23:06:39,773 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:39,774 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46431,1689289576073] 2023-07-13 23:06:39,774 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46431,1689289576073; numProcessing=1 2023-07-13 23:06:39,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:39,775 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,46431,1689289576073 znode expired, triggering replicatorRemoved event 2023-07-13 23:06:39,775 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46431,1689289576073 already deleted, retry=false 2023-07-13 23:06:39,776 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,46431,1689289576073 on jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:39,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:39,777 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=23, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,46431,1689289576073, splitWal=true, meta=true 2023-07-13 23:06:39,777 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=23 for jenkins-hbase4.apache.org,46431,1689289576073 (carryingMeta=true) jenkins-hbase4.apache.org,46431,1689289576073/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@169d9be7[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-13 23:06:39,779 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=23, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46431,1689289576073, splitWal=true, meta=true 2023-07-13 23:06:39,781 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46431,1689289576073, splitWal=true, meta=true, isMeta: true 2023-07-13 23:06:39,792 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting 2023-07-13 23:06:39,794 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,46431,1689289576073 WAL count=1, meta=true 2023-07-13 23:06:39,794 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta}] 2023-07-13 23:06:39,798 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:39,799 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta, worker=jenkins-hbase4.apache.org,46589,1689289582238}] 2023-07-13 23:06:39,858 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:46431 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46431 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 23:06:39,859 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:46431 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46431 2023-07-13 23:06:39,875 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 23:06:39,876 INFO [RS:1;jenkins-hbase4:46431] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,46431,1689289576073; zookeeper connection closed. 2023-07-13 23:06:39,876 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46431-0x10160efc9610002, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 23:06:39,876 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5a994f02] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5a994f02 2023-07-13 23:06:39,955 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:39,956 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-13 23:06:39,957 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-13 23:06:39,958 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46589] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-13 23:06:39,973 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta, size=2.3 K (2403bytes) 2023-07-13 23:06:39,974 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta 2023-07-13 23:06:39,975 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta after 1ms 2023-07-13 23:06:39,979 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta took 6ms 2023-07-13 23:06:39,984 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-13 23:06:39,985 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta so closing down 2023-07-13 23:06:39,985 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-13 23:06:39,990 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta.temp 2023-07-13 23:06:39,990 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-13 23:06:40,006 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) 2023-07-13 23:06:40,008 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta.temp to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-13 23:06:40,008 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 5 edits across 1 Regions in 28 ms; skipped=1; WAL=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false 2023-07-13 23:06:40,008 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta, journal: Splitting hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta, size=2.3 K (2403bytes) at 1689289599974Finishing writing output for hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta so closing down at 1689289599985 (+11 ms)Creating recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta.temp at 1689289599990 (+5 ms)3 split writer threads finished at 1689289599990Closed recovered edits writer path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) at 1689289600006 (+16 ms)Rename recovered edits hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta.temp to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 at 1689289600008 (+2 ms)Processed 5 edits across 1 Regions in 28 ms; skipped=1; WAL=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false at 1689289600008 2023-07-13 23:06:40,008 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta 2023-07-13 23:06:40,008 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-13 23:06:40,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.HMaster(3641): Remote procedure done, pid=25 2023-07-13 23:06:40,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] wal.WALSplitUtil(141): Moved hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs 2023-07-13 23:06:40,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-13 23:06:40,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta, worker=jenkins-hbase4.apache.org,46589,1689289582238 in 214 msec 2023-07-13 23:06:40,016 DEBUG [PEWorker-2] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:40,018 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=23 2023-07-13 23:06:40,018 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C46431%2C1689289576073.meta.1689289580738.meta, worker=jenkins-hbase4.apache.org,46589,1689289582238 in 222 msec 2023-07-13 23:06:40,020 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting dir is empty, no logs to split. 2023-07-13 23:06:40,020 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,46431,1689289576073 WAL count=0, meta=true 2023-07-13 23:06:40,020 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46431,1689289576073 WAL splitting is done? wals=0, meta=true 2023-07-13 23:06:40,021 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-13 23:06:40,022 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-13 23:06:40,023 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 23:06:40,162 DEBUG [hconnection-0x3de720da-metaLookup-shared--pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46431 this server is in the failed servers list 2023-07-13 23:06:40,173 DEBUG [jenkins-hbase4:34895] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-13 23:06:40,174 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46589,1689289582238, state=OPENING 2023-07-13 23:06:40,178 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-13 23:06:40,178 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 23:06:40,178 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46589,1689289582238}] 2023-07-13 23:06:40,337 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-13 23:06:40,338 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-13 23:06:40,340 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46589%2C1689289582238.meta, suffix=.meta, logDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46589,1689289582238, archiveDir=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs, maxLogs=32 2023-07-13 23:06:40,356 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK] 2023-07-13 23:06:40,357 DEBUG [RS-EventLoopGroup-6-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK] 2023-07-13 23:06:40,359 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46589,1689289582238/jenkins-hbase4.apache.org%2C46589%2C1689289582238.meta.1689289600340.meta 2023-07-13 23:06:40,360 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42655,DS-4396672b-9e66-4761-b5f2-06f09255a1b9,DISK], DatanodeInfoWithStorage[127.0.0.1:35445,DS-a5dd6b7b-39c5-4d01-9a7b-f0486077cd15,DISK]] 2023-07-13 23:06:40,360 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:40,360 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-13 23:06:40,360 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-13 23:06:40,360 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-13 23:06:40,360 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-13 23:06:40,360 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:40,360 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-13 23:06:40,360 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-13 23:06:40,362 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-13 23:06:40,363 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info 2023-07-13 23:06:40,363 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info 2023-07-13 23:06:40,364 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-13 23:06:40,373 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info/16f9c5e413d74c93a3d7c19c644a051f 2023-07-13 23:06:40,374 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:40,374 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-13 23:06:40,376 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/rep_barrier 2023-07-13 23:06:40,376 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/rep_barrier 2023-07-13 23:06:40,377 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-13 23:06:40,377 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:40,378 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-13 23:06:40,379 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table 2023-07-13 23:06:40,379 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table 2023-07-13 23:06:40,379 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-13 23:06:40,387 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/table/958f287388b54f1aa4ce9daa93b9a06b 2023-07-13 23:06:40,387 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:40,389 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740 2023-07-13 23:06:40,391 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740 2023-07-13 23:06:40,391 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-13 23:06:40,398 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 16, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=19, path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-13 23:06:40,398 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-13 23:06:40,414 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=19 (bloomFilter=false), to=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/.tmp/info/6df5d1b8ebdb4e30b0b6dca02d7e66f3 2023-07-13 23:06:40,422 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/.tmp/info/6df5d1b8ebdb4e30b0b6dca02d7e66f3 as hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info/6df5d1b8ebdb4e30b0b6dca02d7e66f3 2023-07-13 23:06:40,428 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info/6df5d1b8ebdb4e30b0b6dca02d7e66f3, entries=16, sequenceid=19, filesize=6.4 K 2023-07-13 23:06:40,428 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 30ms, sequenceid=19, compaction requested=false; wal=null 2023-07-13 23:06:40,429 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-13 23:06:40,432 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-13 23:06:40,434 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-13 23:06:40,436 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=13 2023-07-13 23:06:40,437 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=20; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64679868, jitterRate=-0.03619486093521118}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-13 23:06:40,437 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-13 23:06:40,438 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=27, masterSystemTime=1689289600331 2023-07-13 23:06:40,440 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-13 23:06:40,440 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-13 23:06:40,441 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,46589,1689289582238, state=OPEN 2023-07-13 23:06:40,442 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-13 23:06:40,442 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-13 23:06:40,445 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-13 23:06:40,445 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,46589,1689289582238 in 264 msec 2023-07-13 23:06:40,447 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=23 2023-07-13 23:06:40,447 INFO [PEWorker-3] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,46431,1689289576073 had 3 regions 2023-07-13 23:06:40,447 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 424 msec 2023-07-13 23:06:40,448 INFO [PEWorker-3] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,46431,1689289576073, splitWal=true, meta=true, isMeta: false 2023-07-13 23:06:40,450 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,46431,1689289576073 WAL count=1, meta=false 2023-07-13 23:06:40,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425}] 2023-07-13 23:06:40,452 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:40,453 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425, worker=jenkins-hbase4.apache.org,46589,1689289582238}] 2023-07-13 23:06:40,605 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46589] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=29 2023-07-13 23:06:40,619 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425, size=721 (721bytes) 2023-07-13 23:06:40,619 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425 2023-07-13 23:06:40,619 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425 after 0ms 2023-07-13 23:06:40,624 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425 took 5ms 2023-07-13 23:06:40,627 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 5a1d70f57ebb5ff6152a7364d4d54cdd: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-13 23:06:40,628 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 539033c7f1670a68ba89b4a328a12c36: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-13 23:06:40,628 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425 so closing down 2023-07-13 23:06:40,628 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-13 23:06:40,629 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-13 23:06:40,629 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 5 ms; skipped=2; WAL=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425, size=721, length=721, corrupted=false, cancelled=false 2023-07-13 23:06:40,629 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425, journal: Splitting hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425, size=721 (721bytes) at 1689289600619Finishing writing output for hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425 so closing down at 1689289600628 (+9 ms)3 split writer threads finished at 1689289600629 (+1 ms)Processed 2 edits across 0 Regions in 5 ms; skipped=2; WAL=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425, size=721, length=721, corrupted=false, cancelled=false at 1689289600629 2023-07-13 23:06:40,629 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425 2023-07-13 23:06:40,629 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=29 2023-07-13 23:06:40,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] master.HMaster(3641): Remote procedure done, pid=29 2023-07-13 23:06:40,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34895] wal.WALSplitUtil(141): Moved hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting/jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425 to hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs 2023-07-13 23:06:40,635 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-07-13 23:06:40,635 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425, worker=jenkins-hbase4.apache.org,46589,1689289582238 in 179 msec 2023-07-13 23:06:40,636 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:40,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=23 2023-07-13 23:06:40,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C46431%2C1689289576073.1689289577425, worker=jenkins-hbase4.apache.org,46589,1689289582238 in 187 msec 2023-07-13 23:06:40,641 INFO [PEWorker-3] master.SplitLogManager(171): hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/WALs/jenkins-hbase4.apache.org,46431,1689289576073-splitting dir is empty, no logs to split. 2023-07-13 23:06:40,641 INFO [PEWorker-3] master.SplitWALManager(106): jenkins-hbase4.apache.org,46431,1689289576073 WAL count=0, meta=false 2023-07-13 23:06:40,641 DEBUG [PEWorker-3] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,46431,1689289576073 WAL splitting is done? wals=0, meta=false 2023-07-13 23:06:40,643 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN}, {pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN}] 2023-07-13 23:06:40,644 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN 2023-07-13 23:06:40,644 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN 2023-07-13 23:06:40,645 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 23:06:40,645 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-13 23:06:40,664 DEBUG [hconnection-0x3de720da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:40,666 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:40,672 DEBUG [hconnection-0x3de720da-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46431 this server is in the failed servers list 2023-07-13 23:06:40,780 DEBUG [hconnection-0x3de720da-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46431 this server is in the failed servers list 2023-07-13 23:06:40,795 DEBUG [jenkins-hbase4:34895] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-13 23:06:40,796 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=5a1d70f57ebb5ff6152a7364d4d54cdd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:40,796 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=539033c7f1670a68ba89b4a328a12c36, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:40,796 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689289600795"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689289600795"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689289600795"}]},"ts":"1689289600795"} 2023-07-13 23:06:40,796 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689289600795"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689289600795"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689289600795"}]},"ts":"1689289600795"} 2023-07-13 23:06:40,797 WARN [RS-EventLoopGroup-4-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:46431 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46431 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-13 23:06:40,798 DEBUG [RS-EventLoopGroup-4-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:46431 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46431 2023-07-13 23:06:40,899 DEBUG [PEWorker-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-13 23:06:40,901 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-13 23:06:40,903 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=30, state=RUNNABLE; OpenRegionProcedure 539033c7f1670a68ba89b4a328a12c36, server=jenkins-hbase4.apache.org,46589,1689289582238}] 2023-07-13 23:06:40,904 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure 5a1d70f57ebb5ff6152a7364d4d54cdd, server=jenkins-hbase4.apache.org,46589,1689289582238}] 2023-07-13 23:06:40,985 DEBUG [hconnection-0x3de720da-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:46431 this server is in the failed servers list 2023-07-13 23:06:41,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:41,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5a1d70f57ebb5ff6152a7364d4d54cdd, NAME => 't,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:41,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:41,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,063 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,064 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:41,064 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 539033c7f1670a68ba89b4a328a12c36, NAME => 'hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.', STARTKEY => '', ENDKEY => ''} 2023-07-13 23:06:41,064 DEBUG [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f 2023-07-13 23:06:41,065 DEBUG [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f 2023-07-13 23:06:41,065 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:41,065 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-13 23:06:41,065 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:41,065 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5a1d70f57ebb5ff6152a7364d4d54cdd columnFamilyName f 2023-07-13 23:06:41,065 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:41,067 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:41,072 DEBUG [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info 2023-07-13 23:06:41,072 DEBUG [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info 2023-07-13 23:06:41,072 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 539033c7f1670a68ba89b4a328a12c36 columnFamilyName info 2023-07-13 23:06:41,077 DEBUG [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] regionserver.HStore(539): loaded hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/f/d43b3cf4ae4640c18f3ec2e20d787916 2023-07-13 23:06:41,077 INFO [StoreOpener-5a1d70f57ebb5ff6152a7364d4d54cdd-1] regionserver.HStore(310): Store=5a1d70f57ebb5ff6152a7364d4d54cdd/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:41,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,081 DEBUG [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] regionserver.HStore(539): loaded hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/info/35734b3eca764bacae8802aff91a418e 2023-07-13 23:06:41,081 INFO [StoreOpener-539033c7f1670a68ba89b4a328a12c36-1] regionserver.HStore(310): Store=539033c7f1670a68ba89b4a328a12c36/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-13 23:06:41,083 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:41,085 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:41,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-13 23:06:41,089 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:41,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5a1d70f57ebb5ff6152a7364d4d54cdd; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62864882, jitterRate=-0.06324025988578796}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 23:06:41,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5a1d70f57ebb5ff6152a7364d4d54cdd: 2023-07-13 23:06:41,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd., pid=33, masterSystemTime=1689289601055 2023-07-13 23:06:41,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:41,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:41,095 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=5a1d70f57ebb5ff6152a7364d4d54cdd, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:41,095 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-13 23:06:41,096 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689289601095"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689289601095"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689289601095"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689289601095"}]},"ts":"1689289601095"} 2023-07-13 23:06:41,097 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 539033c7f1670a68ba89b4a328a12c36; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59490996, jitterRate=-0.11351507902145386}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-13 23:06:41,097 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 539033c7f1670a68ba89b4a328a12c36: 2023-07-13 23:06:41,098 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36., pid=32, masterSystemTime=1689289601055 2023-07-13 23:06:41,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-07-13 23:06:41,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure 5a1d70f57ebb5ff6152a7364d4d54cdd, server=jenkins-hbase4.apache.org,46589,1689289582238 in 194 msec 2023-07-13 23:06:41,100 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:41,101 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:41,101 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=539033c7f1670a68ba89b4a328a12c36, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:41,101 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689289601101"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689289601101"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689289601101"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689289601101"}]},"ts":"1689289601101"} 2023-07-13 23:06:41,106 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=t, region=5a1d70f57ebb5ff6152a7364d4d54cdd, ASSIGN in 458 msec 2023-07-13 23:06:41,108 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=30 2023-07-13 23:06:41,109 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=30, state=SUCCESS; OpenRegionProcedure 539033c7f1670a68ba89b4a328a12c36, server=jenkins-hbase4.apache.org,46589,1689289582238 in 201 msec 2023-07-13 23:06:41,111 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-07-13 23:06:41,111 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,46431,1689289576073 after splitting done 2023-07-13 23:06:41,111 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=539033c7f1670a68ba89b4a328a12c36, ASSIGN in 466 msec 2023-07-13 23:06:41,111 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase4.apache.org,46431,1689289576073 from processing; numProcessing=0 2023-07-13 23:06:41,113 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,46431,1689289576073, splitWal=true, meta=true in 1.3350 sec 2023-07-13 23:06:41,296 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1103e632 to 127.0.0.1:57722 2023-07-13 23:06:41,296 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:41,297 INFO [Listener at localhost/39491] client.TestScannerTimeout(124): END ************ test2772 2023-07-13 23:06:41,312 INFO [Listener at localhost/39491] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 297), OpenFileDescriptor=577 (was 574) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=139 (was 139), ProcessCount=170 (was 170), AvailableMemoryMB=6914 (was 6924) 2023-07-13 23:06:41,312 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-13 23:06:41,312 INFO [Listener at localhost/39491] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-13 23:06:41,312 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f246984 to 127.0.0.1:57722 2023-07-13 23:06:41,312 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:41,312 DEBUG [Listener at localhost/39491] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-13 23:06:41,313 DEBUG [Listener at localhost/39491] util.JVMClusterUtil(257): Found active master hash=781335908, stopped=false 2023-07-13 23:06:41,313 INFO [Listener at localhost/39491] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:41,315 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-13 23:06:41,315 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-13 23:06:41,315 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:41,315 INFO [Listener at localhost/39491] procedure2.ProcedureExecutor(629): Stopping 2023-07-13 23:06:41,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 23:06:41,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-13 23:06:41,316 DEBUG [Listener at localhost/39491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x515ca995 to 127.0.0.1:57722 2023-07-13 23:06:41,316 DEBUG [Listener at localhost/39491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:41,316 INFO [Listener at localhost/39491] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,46589,1689289582238' ***** 2023-07-13 23:06:41,316 INFO [Listener at localhost/39491] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-13 23:06:41,317 INFO [RS:2;jenkins-hbase4:46589] regionserver.HeapMemoryManager(220): Stopping 2023-07-13 23:06:41,317 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-13 23:06:41,317 INFO [RS:2;jenkins-hbase4:46589] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-13 23:06:41,317 INFO [RS:2;jenkins-hbase4:46589] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-13 23:06:41,317 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(3305): Received CLOSE for 539033c7f1670a68ba89b4a328a12c36 2023-07-13 23:06:41,317 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(3305): Received CLOSE for 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,317 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:41,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 539033c7f1670a68ba89b4a328a12c36, disabling compactions & flushes 2023-07-13 23:06:41,318 DEBUG [RS:2;jenkins-hbase4:46589] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c4a9c37 to 127.0.0.1:57722 2023-07-13 23:06:41,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:41,318 DEBUG [RS:2;jenkins-hbase4:46589] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:41,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:41,318 INFO [RS:2;jenkins-hbase4:46589] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-13 23:06:41,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. after waiting 0 ms 2023-07-13 23:06:41,318 INFO [RS:2;jenkins-hbase4:46589] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-13 23:06:41,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:41,318 INFO [RS:2;jenkins-hbase4:46589] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-13 23:06:41,318 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-13 23:06:41,319 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-13 23:06:41,319 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1478): Online Regions={539033c7f1670a68ba89b4a328a12c36=hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36., 5a1d70f57ebb5ff6152a7364d4d54cdd=t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd., 1588230740=hbase:meta,,1.1588230740} 2023-07-13 23:06:41,319 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-13 23:06:41,319 DEBUG [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1504): Waiting on 1588230740, 539033c7f1670a68ba89b4a328a12c36, 5a1d70f57ebb5ff6152a7364d4d54cdd 2023-07-13 23:06:41,320 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-13 23:06:41,320 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-13 23:06:41,320 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-13 23:06:41,320 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-13 23:06:41,320 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-13 23:06:41,325 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/namespace/539033c7f1670a68ba89b4a328a12c36/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-13 23:06:41,326 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:41,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 539033c7f1670a68ba89b4a328a12c36: 2023-07-13 23:06:41,326 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689289577880.539033c7f1670a68ba89b4a328a12c36. 2023-07-13 23:06:41,327 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5a1d70f57ebb5ff6152a7364d4d54cdd, disabling compactions & flushes 2023-07-13 23:06:41,327 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:41,327 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:41,327 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. after waiting 0 ms 2023-07-13 23:06:41,327 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:41,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/default/t/5a1d70f57ebb5ff6152a7364d4d54cdd/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-13 23:06:41,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:41,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5a1d70f57ebb5ff6152a7364d4d54cdd: 2023-07-13 23:06:41,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689289578657.5a1d70f57ebb5ff6152a7364d4d54cdd. 2023-07-13 23:06:41,349 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=26 (bloomFilter=false), to=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/.tmp/info/0fff585f92cf416d88e673186fa78183 2023-07-13 23:06:41,357 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/.tmp/info/0fff585f92cf416d88e673186fa78183 as hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info/0fff585f92cf416d88e673186fa78183 2023-07-13 23:06:41,364 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/info/0fff585f92cf416d88e673186fa78183, entries=16, sequenceid=26, filesize=6.4 K 2023-07-13 23:06:41,366 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 46ms, sequenceid=26, compaction requested=true 2023-07-13 23:06:41,366 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-13 23:06:41,376 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/data/hbase/meta/1588230740/recovered.edits/29.seqid, newMaxSeqId=29, maxSeqId=19 2023-07-13 23:06:41,377 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-13 23:06:41,377 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-13 23:06:41,377 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-13 23:06:41,377 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-13 23:06:41,520 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,46589,1689289582238; all regions closed. 2023-07-13 23:06:41,526 DEBUG [RS:2;jenkins-hbase4:46589] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs 2023-07-13 23:06:41,527 INFO [RS:2;jenkins-hbase4:46589] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46589%2C1689289582238.meta:.meta(num 1689289600340) 2023-07-13 23:06:41,532 DEBUG [RS:2;jenkins-hbase4:46589] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/oldWALs 2023-07-13 23:06:41,532 INFO [RS:2;jenkins-hbase4:46589] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46589%2C1689289582238:(num 1689289582432) 2023-07-13 23:06:41,532 DEBUG [RS:2;jenkins-hbase4:46589] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:41,532 INFO [RS:2;jenkins-hbase4:46589] regionserver.LeaseManager(133): Closed leases 2023-07-13 23:06:41,532 INFO [RS:2;jenkins-hbase4:46589] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-13 23:06:41,532 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-13 23:06:41,533 INFO [RS:2;jenkins-hbase4:46589] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46589 2023-07-13 23:06:41,536 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46589,1689289582238 2023-07-13 23:06:41,536 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:41,536 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-13 23:06:41,537 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46589,1689289582238] 2023-07-13 23:06:41,537 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46589,1689289582238; numProcessing=1 2023-07-13 23:06:41,538 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46589,1689289582238 already deleted, retry=false 2023-07-13 23:06:41,538 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46589,1689289582238 expired; onlineServers=0 2023-07-13 23:06:41,538 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,34895,1689289574854' ***** 2023-07-13 23:06:41,539 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-13 23:06:41,539 DEBUG [M:0;jenkins-hbase4:34895] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6faf375f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-13 23:06:41,540 INFO [M:0;jenkins-hbase4:34895] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,34895,1689289574854 2023-07-13 23:06:41,540 INFO [M:0;jenkins-hbase4:34895] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,34895,1689289574854; all regions closed. 2023-07-13 23:06:41,540 DEBUG [M:0;jenkins-hbase4:34895] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-13 23:06:41,540 DEBUG [M:0;jenkins-hbase4:34895] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-13 23:06:41,540 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-13 23:06:41,540 DEBUG [M:0;jenkins-hbase4:34895] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-13 23:06:41,540 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689289577092] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689289577092,5,FailOnTimeoutGroup] 2023-07-13 23:06:41,540 INFO [M:0;jenkins-hbase4:34895] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-13 23:06:41,540 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689289577086] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689289577086,5,FailOnTimeoutGroup] 2023-07-13 23:06:41,541 INFO [M:0;jenkins-hbase4:34895] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-13 23:06:41,541 INFO [M:0;jenkins-hbase4:34895] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-13 23:06:41,541 DEBUG [M:0;jenkins-hbase4:34895] master.HMaster(1512): Stopping service threads 2023-07-13 23:06:41,541 INFO [M:0;jenkins-hbase4:34895] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-13 23:06:41,541 ERROR [M:0;jenkins-hbase4:34895] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1124632911) connection to localhost/127.0.0.1:44535 from jenkins,5,PEWorkerGroup] 2023-07-13 23:06:41,541 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-13 23:06:41,542 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-13 23:06:41,542 INFO [M:0;jenkins-hbase4:34895] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-13 23:06:41,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-13 23:06:41,542 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-13 23:06:41,542 DEBUG [M:0;jenkins-hbase4:34895] zookeeper.ZKUtil(398): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-13 23:06:41,543 WARN [M:0;jenkins-hbase4:34895] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-13 23:06:41,543 INFO [M:0;jenkins-hbase4:34895] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-13 23:06:41,543 INFO [M:0;jenkins-hbase4:34895] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-13 23:06:41,543 DEBUG [M:0;jenkins-hbase4:34895] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-13 23:06:41,543 INFO [M:0;jenkins-hbase4:34895] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 23:06:41,543 DEBUG [M:0;jenkins-hbase4:34895] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 23:06:41,543 DEBUG [M:0;jenkins-hbase4:34895] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-13 23:06:41,543 DEBUG [M:0;jenkins-hbase4:34895] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 23:06:41,543 INFO [M:0;jenkins-hbase4:34895] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=111.30 KB heapSize=134.23 KB 2023-07-13 23:06:41,560 INFO [M:0;jenkins-hbase4:34895] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=111.30 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/71b00c8b4f514e72bef37f196eda168c 2023-07-13 23:06:41,567 DEBUG [M:0;jenkins-hbase4:34895] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/71b00c8b4f514e72bef37f196eda168c as hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/71b00c8b4f514e72bef37f196eda168c 2023-07-13 23:06:41,574 INFO [M:0;jenkins-hbase4:34895] regionserver.HStore(1080): Added hdfs://localhost:44535/user/jenkins/test-data/ba06e6cc-fb9a-5ae8-d365-c03bebf48cc3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/71b00c8b4f514e72bef37f196eda168c, entries=33, sequenceid=257, filesize=8.5 K 2023-07-13 23:06:41,575 INFO [M:0;jenkins-hbase4:34895] regionserver.HRegion(2948): Finished flush of dataSize ~111.30 KB/113971, heapSize ~134.22 KB/137440, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=257, compaction requested=false 2023-07-13 23:06:41,576 INFO [M:0;jenkins-hbase4:34895] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-13 23:06:41,577 DEBUG [M:0;jenkins-hbase4:34895] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-13 23:06:41,580 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-13 23:06:41,580 INFO [M:0;jenkins-hbase4:34895] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-13 23:06:41,581 INFO [M:0;jenkins-hbase4:34895] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34895 2023-07-13 23:06:41,583 DEBUG [M:0;jenkins-hbase4:34895] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,34895,1689289574854 already deleted, retry=false 2023-07-13 23:06:41,700 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 23:06:41,700 INFO [M:0;jenkins-hbase4:34895] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,34895,1689289574854; zookeeper connection closed. 2023-07-13 23:06:41,700 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): master:34895-0x10160efc9610000, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 23:06:41,800 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 23:06:41,800 INFO [RS:2;jenkins-hbase4:46589] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,46589,1689289582238; zookeeper connection closed. 2023-07-13 23:06:41,800 DEBUG [Listener at localhost/39491-EventThread] zookeeper.ZKWatcher(600): regionserver:46589-0x10160efc9610008, quorum=127.0.0.1:57722, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-13 23:06:41,801 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5107942c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5107942c 2023-07-13 23:06:41,801 INFO [Listener at localhost/39491] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-13 23:06:41,801 WARN [Listener at localhost/39491] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-13 23:06:41,804 INFO [Listener at localhost/39491] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-13 23:06:41,908 WARN [BP-515644697-172.31.14.131-1689289571841 heartbeating to localhost/127.0.0.1:44535] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-13 23:06:41,909 WARN [BP-515644697-172.31.14.131-1689289571841 heartbeating to localhost/127.0.0.1:44535] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-515644697-172.31.14.131-1689289571841 (Datanode Uuid 30ecc44a-0630-4f97-8054-35640690dc99) service to localhost/127.0.0.1:44535 2023-07-13 23:06:41,910 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/dfs/data/data3/current/BP-515644697-172.31.14.131-1689289571841] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-13 23:06:41,911 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/dfs/data/data4/current/BP-515644697-172.31.14.131-1689289571841] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-13 23:06:41,912 WARN [Listener at localhost/39491] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-13 23:06:41,917 INFO [Listener at localhost/39491] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-13 23:06:42,023 WARN [BP-515644697-172.31.14.131-1689289571841 heartbeating to localhost/127.0.0.1:44535] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-13 23:06:42,023 WARN [BP-515644697-172.31.14.131-1689289571841 heartbeating to localhost/127.0.0.1:44535] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-515644697-172.31.14.131-1689289571841 (Datanode Uuid ac436e4e-ea11-416f-b359-97c9d42b4361) service to localhost/127.0.0.1:44535 2023-07-13 23:06:42,023 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/dfs/data/data1/current/BP-515644697-172.31.14.131-1689289571841] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-13 23:06:42,024 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/66494d90-6cbb-0d70-256a-2cf5ccc4a552/cluster_7dce0f22-0f12-182d-2c7a-8b3ac73c92d6/dfs/data/data2/current/BP-515644697-172.31.14.131-1689289571841] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-13 23:06:42,065 INFO [Listener at localhost/39491] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-13 23:06:42,179 INFO [Listener at localhost/39491] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-13 23:06:42,216 INFO [Listener at localhost/39491] hbase.HBaseTestingUtility(1293): Minicluster is down