2023-07-12 16:05:52,727 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b 2023-07-12 16:05:52,776 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi timeout: 26 mins 2023-07-12 16:05:52,793 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 16:05:52,793 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/cluster_fd62e4bd-2d7f-c592-2cc1-3edac70c12cd, deleteOnExit=true 2023-07-12 16:05:52,794 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 16:05:52,795 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/test.cache.data in system properties and HBase conf 2023-07-12 16:05:52,795 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 16:05:52,796 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/hadoop.log.dir in system properties and HBase conf 2023-07-12 16:05:52,796 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 16:05:52,797 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 16:05:52,797 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 16:05:52,929 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 16:05:53,357 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 16:05:53,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 16:05:53,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 16:05:53,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 16:05:53,362 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 16:05:53,363 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 16:05:53,363 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 16:05:53,363 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 16:05:53,364 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 16:05:53,364 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 16:05:53,365 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/nfs.dump.dir in system properties and HBase conf 2023-07-12 16:05:53,365 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/java.io.tmpdir in system properties and HBase conf 2023-07-12 16:05:53,365 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 16:05:53,366 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 16:05:53,366 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 16:05:53,859 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 16:05:53,864 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 16:05:54,147 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 16:05:54,317 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 16:05:54,331 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 16:05:54,367 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 16:05:54,399 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/java.io.tmpdir/Jetty_localhost_45561_hdfs____.oyc3r/webapp 2023-07-12 16:05:54,526 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45561 2023-07-12 16:05:54,544 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 16:05:54,544 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 16:05:55,109 WARN [Listener at localhost/34197] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 16:05:55,188 WARN [Listener at localhost/34197] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 16:05:55,206 WARN [Listener at localhost/34197] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 16:05:55,213 INFO [Listener at localhost/34197] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 16:05:55,217 INFO [Listener at localhost/34197] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/java.io.tmpdir/Jetty_localhost_35711_datanode____.w7hx9g/webapp 2023-07-12 16:05:55,320 INFO [Listener at localhost/34197] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35711 2023-07-12 16:05:55,605 WARN [Listener at localhost/39041] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 16:05:56,026 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe189eeeb96ef6e65: Processing first storage report for DS-a9468a8e-e9ab-4459-833a-c35bc98c4360 from datanode 6bcaae43-a6a2-463b-91b0-4540258288fe 2023-07-12 16:05:56,027 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe189eeeb96ef6e65: from storage DS-a9468a8e-e9ab-4459-833a-c35bc98c4360 node DatanodeRegistration(127.0.0.1:41675, datanodeUuid=6bcaae43-a6a2-463b-91b0-4540258288fe, infoPort=38639, infoSecurePort=0, ipcPort=39041, storageInfo=lv=-57;cid=testClusterID;nsid=1308254134;c=1689177953937), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-12 16:05:56,027 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xe189eeeb96ef6e65: Processing first storage report for DS-10e076e8-3704-4a2b-a2c2-323fffbd73c5 from datanode 6bcaae43-a6a2-463b-91b0-4540258288fe 2023-07-12 16:05:56,028 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xe189eeeb96ef6e65: from storage DS-10e076e8-3704-4a2b-a2c2-323fffbd73c5 node DatanodeRegistration(127.0.0.1:41675, datanodeUuid=6bcaae43-a6a2-463b-91b0-4540258288fe, infoPort=38639, infoSecurePort=0, ipcPort=39041, storageInfo=lv=-57;cid=testClusterID;nsid=1308254134;c=1689177953937), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 16:05:56,099 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b 2023-07-12 16:05:56,159 INFO [Listener at localhost/39041] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/cluster_fd62e4bd-2d7f-c592-2cc1-3edac70c12cd/zookeeper_0, clientPort=55421, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/cluster_fd62e4bd-2d7f-c592-2cc1-3edac70c12cd/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/cluster_fd62e4bd-2d7f-c592-2cc1-3edac70c12cd/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 16:05:56,172 INFO [Listener at localhost/39041] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=55421 2023-07-12 16:05:56,180 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:56,182 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:56,841 INFO [Listener at localhost/39041] util.FSUtils(471): Created version file at hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9 with version=8 2023-07-12 16:05:56,842 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/hbase-staging 2023-07-12 16:05:57,163 INFO [Listener at localhost/39041] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 16:05:57,676 INFO [Listener at localhost/39041] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=6 2023-07-12 16:05:57,708 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:57,708 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:57,709 INFO [Listener at localhost/39041] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 16:05:57,709 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:57,709 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 16:05:57,854 INFO [Listener at localhost/39041] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 16:05:57,927 DEBUG [Listener at localhost/39041] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 16:05:58,019 INFO [Listener at localhost/39041] ipc.NettyRpcServer(120): Bind to /172.31.10.131:39879 2023-07-12 16:05:58,029 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,031 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,051 INFO [Listener at localhost/39041] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39879 connecting to ZooKeeper ensemble=127.0.0.1:55421 2023-07-12 16:05:58,091 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:398790x0, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 16:05:58,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:39879-0x101c5e4e98d0000 connected 2023-07-12 16:05:58,120 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 16:05:58,120 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 16:05:58,123 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 16:05:58,131 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39879 2023-07-12 16:05:58,131 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39879 2023-07-12 16:05:58,132 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39879 2023-07-12 16:05:58,132 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39879 2023-07-12 16:05:58,132 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39879 2023-07-12 16:05:58,138 INFO [Listener at localhost/39041] master.HMaster(444): hbase.rootdir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9, hbase.cluster.distributed=false 2023-07-12 16:05:58,173 INFO [Listener at localhost/39041] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=6 2023-07-12 16:05:58,174 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,174 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,174 INFO [Listener at localhost/39041] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 16:05:58,174 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,174 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 16:05:58,174 INFO [Listener at localhost/39041] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 16:05:58,175 INFO [Listener at localhost/39041] ipc.NettyRpcServer(120): Bind to /172.31.10.131:34467 2023-07-12 16:05:58,176 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,178 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,180 INFO [Listener at localhost/39041] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34467 connecting to ZooKeeper ensemble=127.0.0.1:55421 2023-07-12 16:05:58,183 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:344670x0, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 16:05:58,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:34467-0x101c5e4e98d0001 connected 2023-07-12 16:05:58,192 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 16:05:58,192 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 16:05:58,193 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 16:05:58,194 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34467 2023-07-12 16:05:58,194 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34467 2023-07-12 16:05:58,194 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34467 2023-07-12 16:05:58,194 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34467 2023-07-12 16:05:58,195 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34467 2023-07-12 16:05:58,195 INFO [Listener at localhost/39041] master.HMaster(444): hbase.rootdir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9, hbase.cluster.distributed=false 2023-07-12 16:05:58,228 INFO [Listener at localhost/39041] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=6 2023-07-12 16:05:58,229 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,229 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,229 INFO [Listener at localhost/39041] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 16:05:58,229 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,229 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 16:05:58,234 INFO [Listener at localhost/39041] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 16:05:58,237 INFO [Listener at localhost/39041] ipc.NettyRpcServer(120): Bind to /172.31.10.131:33647 2023-07-12 16:05:58,239 INFO [Listener at localhost/39041] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 16:05:58,245 DEBUG [Listener at localhost/39041] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 16:05:58,246 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,248 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,249 INFO [Listener at localhost/39041] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33647 connecting to ZooKeeper ensemble=127.0.0.1:55421 2023-07-12 16:05:58,254 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:336470x0, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 16:05:58,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33647-0x101c5e4e98d0002 connected 2023-07-12 16:05:58,255 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 16:05:58,256 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 16:05:58,256 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 16:05:58,257 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33647 2023-07-12 16:05:58,257 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33647 2023-07-12 16:05:58,257 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33647 2023-07-12 16:05:58,258 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33647 2023-07-12 16:05:58,258 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33647 2023-07-12 16:05:58,271 INFO [Listener at localhost/39041] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=6 2023-07-12 16:05:58,271 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,271 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,271 INFO [Listener at localhost/39041] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 16:05:58,272 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 16:05:58,272 INFO [Listener at localhost/39041] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 16:05:58,272 INFO [Listener at localhost/39041] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 16:05:58,273 INFO [Listener at localhost/39041] ipc.NettyRpcServer(120): Bind to /172.31.10.131:38637 2023-07-12 16:05:58,273 INFO [Listener at localhost/39041] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 16:05:58,276 DEBUG [Listener at localhost/39041] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 16:05:58,276 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,279 INFO [Listener at localhost/39041] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,281 INFO [Listener at localhost/39041] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38637 connecting to ZooKeeper ensemble=127.0.0.1:55421 2023-07-12 16:05:58,285 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:386370x0, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 16:05:58,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:38637-0x101c5e4e98d0003 connected 2023-07-12 16:05:58,286 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 16:05:58,287 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 16:05:58,288 DEBUG [Listener at localhost/39041] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 16:05:58,288 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38637 2023-07-12 16:05:58,289 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38637 2023-07-12 16:05:58,289 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38637 2023-07-12 16:05:58,289 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38637 2023-07-12 16:05:58,290 DEBUG [Listener at localhost/39041] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38637 2023-07-12 16:05:58,291 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,34467,1689177958173 2023-07-12 16:05:58,292 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:05:58,301 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 16:05:58,301 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 16:05:58,303 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,34467,1689177958173 2023-07-12 16:05:58,303 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:05:58,327 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 16:05:58,328 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 16:05:58,328 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 16:05:58,327 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 16:05:58,328 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:05:58,328 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:05:58,329 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 16:05:58,330 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 16:05:58,331 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,39879,1689177956989 from backup master directory 2023-07-12 16:05:58,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 16:05:58,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 16:05:58,331 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase5.apache.org,39879,1689177956989; waiting to become the next active master 2023-07-12 16:05:58,336 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:05:58,336 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 16:05:58,336 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 16:05:58,337 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 16:05:58,337 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:05:58,340 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 16:05:58,341 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 16:05:58,839 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/hbase.id with ID: e8c1355b-ab41-4227-8916-8a1fb5813116 2023-07-12 16:05:58,885 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 16:05:58,899 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:05:58,899 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:05:59,340 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x5892ba3f to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:05:59,374 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58b7fb42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:05:59,398 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:05:59,400 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 16:05:59,417 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 16:05:59,417 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 16:05:59,419 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 16:05:59,423 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 16:05:59,424 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 16:05:59,456 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store-tmp 2023-07-12 16:05:59,883 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:05:59,883 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 16:05:59,883 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 16:05:59,883 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 16:05:59,883 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 16:05:59,883 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 16:05:59,883 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 16:05:59,884 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 16:05:59,885 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/WALs/jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:05:59,906 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C39879%2C1689177956989, suffix=, logDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/WALs/jenkins-hbase5.apache.org,39879,1689177956989, archiveDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/oldWALs, maxLogs=10 2023-07-12 16:05:59,960 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41675,DS-a9468a8e-e9ab-4459-833a-c35bc98c4360,DISK] 2023-07-12 16:05:59,967 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 16:06:00,025 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/WALs/jenkins-hbase5.apache.org,39879,1689177956989/jenkins-hbase5.apache.org%2C39879%2C1689177956989.1689177959916 2023-07-12 16:06:00,026 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41675,DS-a9468a8e-e9ab-4459-833a-c35bc98c4360,DISK]] 2023-07-12 16:06:00,026 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:00,027 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:00,030 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 16:06:00,031 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 16:06:00,095 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 16:06:00,101 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 16:06:00,126 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 16:06:00,138 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:00,143 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 16:06:00,145 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 16:06:00,160 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 16:06:00,164 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:06:00,165 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65897693, jitterRate=-0.01804785430431366}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:00,165 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 16:06:00,166 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 16:06:00,185 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 16:06:00,185 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 16:06:00,187 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 16:06:00,189 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 16:06:00,220 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-12 16:06:00,220 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 16:06:00,245 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 16:06:00,250 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 16:06:00,274 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 16:06:00,277 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 16:06:00,278 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 16:06:00,283 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 16:06:00,287 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 16:06:00,291 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:00,291 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:00,292 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 16:06:00,293 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 16:06:00,303 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 16:06:00,308 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 16:06:00,308 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 16:06:00,308 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:00,308 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 16:06:00,308 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 16:06:00,308 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:00,308 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,39879,1689177956989, sessionid=0x101c5e4e98d0000, setting cluster-up flag (Was=false) 2023-07-12 16:06:00,311 INFO [M:1;jenkins-hbase5:34467] regionserver.HRegionServer(951): ClusterId : e8c1355b-ab41-4227-8916-8a1fb5813116 2023-07-12 16:06:00,323 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:00,323 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:00,329 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 16:06:00,330 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:06:00,336 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:00,336 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:00,342 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 16:06:00,343 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:06:00,346 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp 2023-07-12 16:06:00,394 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(951): ClusterId : e8c1355b-ab41-4227-8916-8a1fb5813116 2023-07-12 16:06:00,395 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(951): ClusterId : e8c1355b-ab41-4227-8916-8a1fb5813116 2023-07-12 16:06:00,400 DEBUG [RS:1;jenkins-hbase5:38637] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 16:06:00,400 DEBUG [RS:0;jenkins-hbase5:33647] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 16:06:00,410 DEBUG [RS:0;jenkins-hbase5:33647] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 16:06:00,410 DEBUG [RS:1;jenkins-hbase5:38637] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 16:06:00,410 DEBUG [RS:0;jenkins-hbase5:33647] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 16:06:00,410 DEBUG [RS:1;jenkins-hbase5:38637] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 16:06:00,413 DEBUG [RS:0;jenkins-hbase5:33647] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 16:06:00,413 DEBUG [RS:1;jenkins-hbase5:38637] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 16:06:00,414 DEBUG [RS:0;jenkins-hbase5:33647] zookeeper.ReadOnlyZKClient(139): Connect 0x12dde343 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:00,415 DEBUG [RS:1;jenkins-hbase5:38637] zookeeper.ReadOnlyZKClient(139): Connect 0x21a06c2b to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:00,422 DEBUG [RS:0;jenkins-hbase5:33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c1a0f68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:00,423 DEBUG [RS:0;jenkins-hbase5:33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18258833, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 16:06:00,423 DEBUG [RS:1;jenkins-hbase5:38637] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f67866f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:00,423 DEBUG [RS:1;jenkins-hbase5:38637] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b58075b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 16:06:00,446 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase5:33647 2023-07-12 16:06:00,447 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase5:38637 2023-07-12 16:06:00,450 INFO [RS:0;jenkins-hbase5:33647] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 16:06:00,450 INFO [RS:1;jenkins-hbase5:38637] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 16:06:00,451 INFO [RS:0;jenkins-hbase5:33647] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 16:06:00,451 INFO [RS:1;jenkins-hbase5:38637] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 16:06:00,451 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 16:06:00,451 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 16:06:00,454 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,39879,1689177956989 with isa=jenkins-hbase5.apache.org/172.31.10.131:38637, startcode=1689177958270 2023-07-12 16:06:00,454 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,39879,1689177956989 with isa=jenkins-hbase5.apache.org/172.31.10.131:33647, startcode=1689177958228 2023-07-12 16:06:00,473 DEBUG [RS:1;jenkins-hbase5:38637] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 16:06:00,473 DEBUG [RS:0;jenkins-hbase5:33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 16:06:00,475 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 16:06:00,484 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 16:06:00,485 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 16:06:00,485 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 16:06:00,485 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 16:06:00,485 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 16:06:00,485 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,485 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 16:06:00,485 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,487 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689177990487 2023-07-12 16:06:00,490 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 16:06:00,493 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 16:06:00,493 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 16:06:00,502 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 16:06:00,505 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 16:06:00,513 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 16:06:00,514 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 16:06:00,514 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 16:06:00,514 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 16:06:00,515 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,516 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 16:06:00,518 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 16:06:00,518 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 16:06:00,520 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 16:06:00,521 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 16:06:00,523 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689177960522,5,FailOnTimeoutGroup] 2023-07-12 16:06:00,523 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689177960523,5,FailOnTimeoutGroup] 2023-07-12 16:06:00,523 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,524 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 16:06:00,525 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,526 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,537 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:53129, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 16:06:00,537 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:47213, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 16:06:00,538 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:00,539 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:00,539 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9 2023-07-12 16:06:00,551 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39879] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:00,561 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39879] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:00,566 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9 2023-07-12 16:06:00,566 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9 2023-07-12 16:06:00,567 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:34197 2023-07-12 16:06:00,567 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:34197 2023-07-12 16:06:00,567 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 16:06:00,567 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 16:06:00,575 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 16:06:00,576 DEBUG [RS:1;jenkins-hbase5:38637] zookeeper.ZKUtil(162): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:00,576 DEBUG [RS:0;jenkins-hbase5:33647] zookeeper.ZKUtil(162): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:00,576 WARN [RS:1;jenkins-hbase5:38637] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 16:06:00,576 WARN [RS:0;jenkins-hbase5:33647] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 16:06:00,576 INFO [RS:1;jenkins-hbase5:38637] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 16:06:00,577 INFO [RS:0;jenkins-hbase5:33647] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 16:06:00,577 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1948): logDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:00,577 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1948): logDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:00,577 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,33647,1689177958228] 2023-07-12 16:06:00,577 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,38637,1689177958270] 2023-07-12 16:06:00,587 DEBUG [RS:1;jenkins-hbase5:38637] zookeeper.ZKUtil(162): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:00,587 DEBUG [RS:0;jenkins-hbase5:33647] zookeeper.ZKUtil(162): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:00,587 DEBUG [RS:1;jenkins-hbase5:38637] zookeeper.ZKUtil(162): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:00,587 DEBUG [RS:0;jenkins-hbase5:33647] zookeeper.ZKUtil(162): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:00,597 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 16:06:00,597 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 16:06:00,606 INFO [RS:1;jenkins-hbase5:38637] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 16:06:00,606 INFO [RS:0;jenkins-hbase5:33647] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 16:06:00,627 INFO [RS:1;jenkins-hbase5:38637] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 16:06:00,627 INFO [RS:0;jenkins-hbase5:33647] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 16:06:00,631 INFO [RS:1;jenkins-hbase5:38637] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 16:06:00,631 INFO [RS:0;jenkins-hbase5:33647] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 16:06:00,631 INFO [RS:1;jenkins-hbase5:38637] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,631 INFO [RS:0;jenkins-hbase5:33647] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,632 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 16:06:00,632 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 16:06:00,639 INFO [RS:1;jenkins-hbase5:38637] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,639 INFO [RS:0;jenkins-hbase5:33647] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,640 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,640 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,640 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,640 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,640 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,640 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,640 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,640 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,640 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 16:06:00,641 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 16:06:00,641 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:1;jenkins-hbase5:38637] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,641 DEBUG [RS:0;jenkins-hbase5:33647] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 16:06:00,643 INFO [RS:0;jenkins-hbase5:33647] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,643 INFO [RS:1;jenkins-hbase5:38637] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,643 INFO [RS:0;jenkins-hbase5:33647] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,643 INFO [RS:1;jenkins-hbase5:38637] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,643 INFO [RS:0;jenkins-hbase5:33647] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,643 INFO [RS:1;jenkins-hbase5:38637] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,659 INFO [RS:0;jenkins-hbase5:33647] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 16:06:00,659 INFO [RS:1;jenkins-hbase5:38637] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 16:06:00,662 INFO [RS:0;jenkins-hbase5:33647] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,33647,1689177958228-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,662 INFO [RS:1;jenkins-hbase5:38637] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,38637,1689177958270-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:00,679 INFO [RS:0;jenkins-hbase5:33647] regionserver.Replication(203): jenkins-hbase5.apache.org,33647,1689177958228 started 2023-07-12 16:06:00,679 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,33647,1689177958228, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:33647, sessionid=0x101c5e4e98d0002 2023-07-12 16:06:00,679 DEBUG [RS:0;jenkins-hbase5:33647] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 16:06:00,679 DEBUG [RS:0;jenkins-hbase5:33647] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:00,680 DEBUG [RS:0;jenkins-hbase5:33647] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,33647,1689177958228' 2023-07-12 16:06:00,680 DEBUG [RS:0;jenkins-hbase5:33647] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 16:06:00,680 DEBUG [RS:0;jenkins-hbase5:33647] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 16:06:00,681 DEBUG [RS:0;jenkins-hbase5:33647] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 16:06:00,681 DEBUG [RS:0;jenkins-hbase5:33647] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 16:06:00,681 DEBUG [RS:0;jenkins-hbase5:33647] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:00,681 DEBUG [RS:0;jenkins-hbase5:33647] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,33647,1689177958228' 2023-07-12 16:06:00,681 DEBUG [RS:0;jenkins-hbase5:33647] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:00,682 DEBUG [RS:0;jenkins-hbase5:33647] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:00,682 DEBUG [RS:0;jenkins-hbase5:33647] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 16:06:00,683 INFO [RS:0;jenkins-hbase5:33647] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 16:06:00,683 INFO [RS:1;jenkins-hbase5:38637] regionserver.Replication(203): jenkins-hbase5.apache.org,38637,1689177958270 started 2023-07-12 16:06:00,683 INFO [RS:0;jenkins-hbase5:33647] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 16:06:00,683 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,38637,1689177958270, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:38637, sessionid=0x101c5e4e98d0003 2023-07-12 16:06:00,683 DEBUG [RS:1;jenkins-hbase5:38637] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 16:06:00,683 DEBUG [RS:1;jenkins-hbase5:38637] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:00,684 DEBUG [RS:1;jenkins-hbase5:38637] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,38637,1689177958270' 2023-07-12 16:06:00,684 DEBUG [RS:1;jenkins-hbase5:38637] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 16:06:00,684 DEBUG [RS:1;jenkins-hbase5:38637] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 16:06:00,685 DEBUG [RS:1;jenkins-hbase5:38637] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 16:06:00,685 DEBUG [RS:1;jenkins-hbase5:38637] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 16:06:00,685 DEBUG [RS:1;jenkins-hbase5:38637] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:00,685 DEBUG [RS:1;jenkins-hbase5:38637] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,38637,1689177958270' 2023-07-12 16:06:00,685 DEBUG [RS:1;jenkins-hbase5:38637] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:00,686 DEBUG [RS:1;jenkins-hbase5:38637] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:00,686 DEBUG [RS:1;jenkins-hbase5:38637] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 16:06:00,686 INFO [RS:1;jenkins-hbase5:38637] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 16:06:00,687 INFO [RS:1;jenkins-hbase5:38637] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 16:06:00,794 INFO [RS:0;jenkins-hbase5:33647] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C33647%2C1689177958228, suffix=, logDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,33647,1689177958228, archiveDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/oldWALs, maxLogs=32 2023-07-12 16:06:00,794 INFO [RS:1;jenkins-hbase5:38637] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C38637%2C1689177958270, suffix=, logDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,38637,1689177958270, archiveDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/oldWALs, maxLogs=32 2023-07-12 16:06:00,820 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41675,DS-a9468a8e-e9ab-4459-833a-c35bc98c4360,DISK] 2023-07-12 16:06:00,821 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41675,DS-a9468a8e-e9ab-4459-833a-c35bc98c4360,DISK] 2023-07-12 16:06:00,826 INFO [RS:1;jenkins-hbase5:38637] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,38637,1689177958270/jenkins-hbase5.apache.org%2C38637%2C1689177958270.1689177960797 2023-07-12 16:06:00,826 INFO [RS:0;jenkins-hbase5:33647] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,33647,1689177958228/jenkins-hbase5.apache.org%2C33647%2C1689177958228.1689177960797 2023-07-12 16:06:00,826 DEBUG [RS:1;jenkins-hbase5:38637] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41675,DS-a9468a8e-e9ab-4459-833a-c35bc98c4360,DISK]] 2023-07-12 16:06:00,828 DEBUG [RS:0;jenkins-hbase5:33647] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41675,DS-a9468a8e-e9ab-4459-833a-c35bc98c4360,DISK]] 2023-07-12 16:06:00,954 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:00,957 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 16:06:00,959 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/info 2023-07-12 16:06:00,960 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 16:06:00,961 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:00,961 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 16:06:00,963 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/rep_barrier 2023-07-12 16:06:00,964 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 16:06:00,965 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:00,965 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 16:06:00,967 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/table 2023-07-12 16:06:00,967 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 16:06:00,968 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:00,970 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740 2023-07-12 16:06:00,971 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740 2023-07-12 16:06:00,974 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 16:06:00,976 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 16:06:00,979 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:06:00,980 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73220550, jitterRate=0.091071218252182}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 16:06:00,980 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 16:06:00,980 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 16:06:00,980 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 16:06:00,980 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 16:06:00,980 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 16:06:00,980 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 16:06:00,981 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 16:06:00,981 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 16:06:00,986 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 16:06:00,986 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 16:06:00,995 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 16:06:01,010 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 16:06:01,012 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 16:06:01,164 DEBUG [jenkins-hbase5:39879] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-12 16:06:01,168 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:01,173 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:01,173 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:01,173 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:01,176 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,38637,1689177958270, state=OPENING 2023-07-12 16:06:01,183 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 16:06:01,185 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:01,185 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:01,186 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 16:06:01,186 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 16:06:01,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:06:01,378 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:01,381 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 16:06:01,384 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:41812, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 16:06:01,396 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 16:06:01,397 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 16:06:01,401 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C38637%2C1689177958270.meta, suffix=.meta, logDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,38637,1689177958270, archiveDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/oldWALs, maxLogs=32 2023-07-12 16:06:01,420 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41675,DS-a9468a8e-e9ab-4459-833a-c35bc98c4360,DISK] 2023-07-12 16:06:01,424 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,38637,1689177958270/jenkins-hbase5.apache.org%2C38637%2C1689177958270.meta.1689177961403.meta 2023-07-12 16:06:01,424 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41675,DS-a9468a8e-e9ab-4459-833a-c35bc98c4360,DISK]] 2023-07-12 16:06:01,425 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:01,427 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 16:06:01,445 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 16:06:01,451 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 16:06:01,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 16:06:01,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:01,457 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 16:06:01,457 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 16:06:01,459 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 16:06:01,462 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/info 2023-07-12 16:06:01,462 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/info 2023-07-12 16:06:01,463 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 16:06:01,465 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:01,465 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 16:06:01,468 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/rep_barrier 2023-07-12 16:06:01,468 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/rep_barrier 2023-07-12 16:06:01,468 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 16:06:01,469 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:01,470 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 16:06:01,471 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/table 2023-07-12 16:06:01,471 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/table 2023-07-12 16:06:01,472 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 16:06:01,473 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:01,475 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740 2023-07-12 16:06:01,477 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740 2023-07-12 16:06:01,481 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 16:06:01,484 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 16:06:01,485 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65970205, jitterRate=-0.01696734130382538}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 16:06:01,485 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 16:06:01,495 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689177961371 2023-07-12 16:06:01,512 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 16:06:01,513 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 16:06:01,513 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,38637,1689177958270, state=OPEN 2023-07-12 16:06:01,516 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 16:06:01,516 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 16:06:01,517 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 16:06:01,517 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 16:06:01,521 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 16:06:01,521 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,38637,1689177958270 in 328 msec 2023-07-12 16:06:01,526 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 16:06:01,526 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 527 msec 2023-07-12 16:06:01,531 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1280 sec 2023-07-12 16:06:01,531 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689177961531, completionTime=-1 2023-07-12 16:06:01,532 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-12 16:06:01,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 16:06:01,603 DEBUG [hconnection-0x49280002-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:01,605 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:41828, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:01,625 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-12 16:06:01,625 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689178021625 2023-07-12 16:06:01,625 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689178081625 2023-07-12 16:06:01,625 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 92 msec 2023-07-12 16:06:01,654 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39879,1689177956989-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:01,654 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39879,1689177956989-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:01,654 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39879,1689177956989-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:01,656 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:39879, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:01,656 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 16:06:01,663 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 16:06:01,672 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 16:06:01,674 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 16:06:01,685 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 16:06:01,687 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:06:01,691 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:06:01,713 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:01,715 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65 empty. 2023-07-12 16:06:01,716 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:01,716 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 16:06:01,744 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:01,747 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6527c424ba7b867664c691dcf6c1ec65, NAME => 'hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:06:02,161 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:02,162 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 6527c424ba7b867664c691dcf6c1ec65, disabling compactions & flushes 2023-07-12 16:06:02,162 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:06:02,162 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:06:02,162 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. after waiting 0 ms 2023-07-12 16:06:02,162 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:06:02,162 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:06:02,162 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 6527c424ba7b867664c691dcf6c1ec65: 2023-07-12 16:06:02,168 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:06:02,186 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689177962171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177962171"}]},"ts":"1689177962171"} 2023-07-12 16:06:02,213 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:02,215 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:06:02,219 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177962215"}]},"ts":"1689177962215"} 2023-07-12 16:06:02,222 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 16:06:02,226 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:02,227 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:02,227 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:02,227 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:02,230 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6527c424ba7b867664c691dcf6c1ec65, ASSIGN}] 2023-07-12 16:06:02,232 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6527c424ba7b867664c691dcf6c1ec65, ASSIGN 2023-07-12 16:06:02,233 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=6527c424ba7b867664c691dcf6c1ec65, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,33647,1689177958228; forceNewPlan=false, retain=false 2023-07-12 16:06:02,386 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:06:02,387 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6527c424ba7b867664c691dcf6c1ec65, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:02,387 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689177962387"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177962387"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177962387"}]},"ts":"1689177962387"} 2023-07-12 16:06:02,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 6527c424ba7b867664c691dcf6c1ec65, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:02,547 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:02,547 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 16:06:02,550 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43240, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 16:06:02,555 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:06:02,556 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6527c424ba7b867664c691dcf6c1ec65, NAME => 'hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:02,556 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:02,556 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:02,556 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:02,557 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:02,559 INFO [StoreOpener-6527c424ba7b867664c691dcf6c1ec65-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:02,561 DEBUG [StoreOpener-6527c424ba7b867664c691dcf6c1ec65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65/info 2023-07-12 16:06:02,561 DEBUG [StoreOpener-6527c424ba7b867664c691dcf6c1ec65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65/info 2023-07-12 16:06:02,562 INFO [StoreOpener-6527c424ba7b867664c691dcf6c1ec65-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6527c424ba7b867664c691dcf6c1ec65 columnFamilyName info 2023-07-12 16:06:02,562 INFO [StoreOpener-6527c424ba7b867664c691dcf6c1ec65-1] regionserver.HStore(310): Store=6527c424ba7b867664c691dcf6c1ec65/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:02,564 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:02,565 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:02,569 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:06:02,572 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:06:02,573 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 6527c424ba7b867664c691dcf6c1ec65; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61648364, jitterRate=-0.08136779069900513}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:02,573 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 6527c424ba7b867664c691dcf6c1ec65: 2023-07-12 16:06:02,575 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65., pid=6, masterSystemTime=1689177962547 2023-07-12 16:06:02,579 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:06:02,579 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:06:02,581 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6527c424ba7b867664c691dcf6c1ec65, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:02,581 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689177962580"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689177962580"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689177962580"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689177962580"}]},"ts":"1689177962580"} 2023-07-12 16:06:02,587 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 16:06:02,588 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 6527c424ba7b867664c691dcf6c1ec65, server=jenkins-hbase5.apache.org,33647,1689177958228 in 192 msec 2023-07-12 16:06:02,591 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 16:06:02,591 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=6527c424ba7b867664c691dcf6c1ec65, ASSIGN in 358 msec 2023-07-12 16:06:02,592 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:06:02,593 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177962593"}]},"ts":"1689177962593"} 2023-07-12 16:06:02,595 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 16:06:02,599 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:06:02,602 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 925 msec 2023-07-12 16:06:02,688 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 16:06:02,689 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 16:06:02,689 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:02,689 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:06:02,711 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:02,714 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:02,731 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 16:06:02,745 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 16:06:02,751 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-07-12 16:06:02,763 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 16:06:02,774 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 16:06:02,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-07-12 16:06:02,789 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 16:06:02,797 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 16:06:02,797 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.460sec 2023-07-12 16:06:02,800 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 16:06:02,801 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 16:06:02,801 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 16:06:02,802 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39879,1689177956989-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 16:06:02,803 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,39879,1689177956989-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 16:06:02,811 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 16:06:02,901 DEBUG [Listener at localhost/39041] zookeeper.ReadOnlyZKClient(139): Connect 0x6b9833e1 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:02,906 DEBUG [Listener at localhost/39041] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d8a2e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:02,941 DEBUG [hconnection-0x168f361e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:02,953 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:41838, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:02,962 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:06:02,963 DEBUG [Listener at localhost/39041] zookeeper.ReadOnlyZKClient(139): Connect 0x6b24fef6 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:02,972 DEBUG [ReadOnlyZKClient-127.0.0.1:55421@0x6b24fef6] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@aa0efa7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:02,999 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testRestoreSnapshot[0] Thread=265, OpenFileDescriptor=528, MaxFileDescriptor=60000, SystemLoadAverage=166, ProcessCount=161, AvailableMemoryMB=7367 2023-07-12 16:06:03,039 DEBUG [Listener at localhost/39041] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 16:06:03,042 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49594, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 16:06:03,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testRestoreSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:06:03,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:03,058 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:06:03,060 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:06:03,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot" procId is: 9 2023-07-12 16:06:03,063 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:03,064 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 empty. 2023-07-12 16:06:03,066 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:03,066 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot regions 2023-07-12 16:06:03,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 16:06:03,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 16:06:03,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 16:06:03,484 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:03,485 INFO [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:06:03,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 16:06:03,898 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:03,898 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1604): Closing 7def597ffc2b8cfcdf545c2c6ab26304, disabling compactions & flushes 2023-07-12 16:06:03,899 INFO [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:03,899 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:03,899 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. after waiting 0 ms 2023-07-12 16:06:03,899 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:03,899 INFO [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1838): Closed testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:03,899 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:03,903 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:06:03,905 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177963904"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177963904"}]},"ts":"1689177963904"} 2023-07-12 16:06:03,907 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:03,908 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:06:03,908 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177963908"}]},"ts":"1689177963908"} 2023-07-12 16:06:03,911 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:06:03,919 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:03,920 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:03,920 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:03,920 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:03,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN}] 2023-07-12 16:06:03,922 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN 2023-07-12 16:06:03,924 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,33647,1689177958228; forceNewPlan=false, retain=false 2023-07-12 16:06:04,074 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:06:04,075 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:04,075 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177964075"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177964075"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177964075"}]},"ts":"1689177964075"} 2023-07-12 16:06:04,078 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:04,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 16:06:04,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:04,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:04,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:04,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:04,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:04,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:04,239 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:04,241 DEBUG [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1 2023-07-12 16:06:04,241 DEBUG [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1 2023-07-12 16:06:04,242 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7def597ffc2b8cfcdf545c2c6ab26304 columnFamilyName f1 2023-07-12 16:06:04,243 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] regionserver.HStore(310): Store=7def597ffc2b8cfcdf545c2c6ab26304/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:04,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:04,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:04,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:04,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:06:04,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 7def597ffc2b8cfcdf545c2c6ab26304; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68852116, jitterRate=0.025976479053497314}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:04,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:04,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304., pid=11, masterSystemTime=1689177964231 2023-07-12 16:06:04,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:04,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:04,258 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:04,258 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177964258"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689177964258"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689177964258"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689177964258"}]},"ts":"1689177964258"} 2023-07-12 16:06:04,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 16:06:04,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228 in 183 msec 2023-07-12 16:06:04,266 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 16:06:04,266 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN in 343 msec 2023-07-12 16:06:04,267 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:06:04,268 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177964267"}]},"ts":"1689177964267"} 2023-07-12 16:06:04,270 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:06:04,273 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:06:04,275 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot in 1.2180 sec 2023-07-12 16:06:05,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-12 16:06:05,187 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot, procId: 9 completed 2023-07-12 16:06:05,187 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testRestoreSnapshot get assigned. Timeout = 60000ms 2023-07-12 16:06:05,189 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:05,194 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testRestoreSnapshot assigned to meta. Checking AM states. 2023-07-12 16:06:05,195 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:05,195 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testRestoreSnapshot assigned. 2023-07-12 16:06:05,198 DEBUG [Listener at localhost/39041] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:05,201 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:06,986 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:06:07,076 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 16:06:07,078 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot' 2023-07-12 16:06:07,079 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 16:06:07,159 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 16:06:07,160 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 16:06:07,614 DEBUG [Listener at localhost/39041] client.ConnectionUtils(586): Start fetching master stub from registry 2023-07-12 16:06:07,616 DEBUG [ReadOnlyZKClient-127.0.0.1:55421@0x6b24fef6] client.AsyncConnectionImpl(289): The fetched master address is jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:06:07,619 DEBUG [ReadOnlyZKClient-127.0.0.1:55421@0x6b24fef6] client.ConnectionUtils(594): The fetched master stub is org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$Stub@146db49d 2023-07-12 16:06:07,624 DEBUG [ReadOnlyZKClient-127.0.0.1:55421@0x6b24fef6] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 16:06:07,627 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:39656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 16:06:07,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:07,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689177967641 (current time:1689177967641). 2023-07-12 16:06:07,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:07,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x07b55f7c to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:07,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ad7720e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:07,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:07,652 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:40204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:07,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07b55f7c to 127.0.0.1:55421 2023-07-12 16:06:07,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:07,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:07,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:07,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:07,674 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=12, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:07,674 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=12, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178567674 2023-07-12 16:06:07,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:07,675 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot 2023-07-12 16:06:07,676 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=12, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:07,677 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=13, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:06:07,677 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE in 10 msec 2023-07-12 16:06:07,678 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=13, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:06:07,678 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=13, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED; timeout=600000, timestamp=1689178567678 2023-07-12 16:06:07,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:07,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:07,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:07,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:08,100 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:06:08,100 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:06:08,100 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:08,101 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:06:08,101 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:06:08,101 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:08,101 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:08,103 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:08,103 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:08,103 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:08,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:08,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:08,104 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:08,104 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:08,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:08,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:08,105 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testRestoreSnapshot type FLUSH 2023-07-12 16:06:08,105 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testRestoreSnapshot type FLUSH 2023-07-12 16:06:08,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:08,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:08,108 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:08,108 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:08,108 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:08,108 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:08,108 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:08,109 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:08,109 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:08,109 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:08,109 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:08,109 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:08,112 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:08,112 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:08,113 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:08,113 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,114 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:08,114 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,114 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:08,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:06:08,116 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:06:08,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317deb1b[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:08,116 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,117 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,117 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:08,118 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:08,118 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:08,118 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:08,118 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:08,118 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:08,118 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:08,119 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:08,119 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:08,119 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:08,119 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. started... 2023-07-12 16:06:08,121 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:08,121 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:08,121 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:08,122 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2745): Flushing 7def597ffc2b8cfcdf545c2c6ab26304 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:06:08,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:08,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:08,626 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/.tmp/f1/61528cc5de4648dba74da5eccc2292e8 2023-07-12 16:06:08,669 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/.tmp/f1/61528cc5de4648dba74da5eccc2292e8 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8 2023-07-12 16:06:08,678 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:06:08,680 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for 7def597ffc2b8cfcdf545c2c6ab26304 in 558ms, sequenceid=3004, compaction requested=false 2023-07-12 16:06:08,682 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:08,682 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.' region-info for snapshot=snapshotName1 2023-07-12 16:06:08,687 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:08,691 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8] hfiles 2023-07-12 16:06:08,691 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8 for snapshot=snapshotName1 2023-07-12 16:06:08,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:08,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:09,105 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. completed. 2023-07-12 16:06:09,106 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:09,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:09,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:09,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:09,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:09,109 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:09,109 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,109 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,110 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:06:09,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:06:09,115 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:06:09,115 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:09,115 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:09,116 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:06:09,116 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:09,118 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,118 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,119 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:09,119 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:09,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,118 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:09,119 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:09,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,120 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:09,122 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:09,123 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:09,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:09,134 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:06:09,134 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:09,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:09,135 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:06:09,134 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:09,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:09,135 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:09,135 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:09,135 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:09,135 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:09,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:09,138 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:09,573 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:06:09,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:09,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:10,407 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testRestoreSnapshot completed 2023-07-12 16:06:10,407 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:10,407 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:10,408 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot at 1689177967675Consolidate snapshot: snapshotName1 at 1689177969135 (+1460 ms)Loading Region manifests for snapshotName1 at 1689177969135Writing data manifest for snapshotName1 at 1689177969146 (+11 ms)Verifying snapshot: snapshotName1 at 1689177969561 (+415 ms)Snapshot snapshotName1 of table testRestoreSnapshot completed at 1689177970407 (+846 ms) 2023-07-12 16:06:10,409 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=13, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:06:10,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED in 2.7330 sec 2023-07-12 16:06:11,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:11,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:11,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:11,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689177971741 (current time:1689177971741). 2023-07-12 16:06:11,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:11,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x4e5ada6d to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:11,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25adf375, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:11,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:11,748 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:40216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:11,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e5ada6d to 127.0.0.1:55421 2023-07-12 16:06:11,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:11,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:11,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:11,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=14, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:11,754 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=14, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:11,754 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=14, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178571754 2023-07-12 16:06:11,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:11,754 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot 2023-07-12 16:06:11,755 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=14, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:11,756 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:06:11,756 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE in 3 msec 2023-07-12 16:06:11,757 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=15, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:06:11,757 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=15, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED; timeout=600000, timestamp=1689178571757 2023-07-12 16:06:11,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:11,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:11,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:11,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:12,170 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName2 2023-07-12 16:06:12,170 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName2' 2023-07-12 16:06:12,170 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:12,170 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName2' starting 'acquire' 2023-07-12 16:06:12,170 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName2', kicking off acquire phase on members. 2023-07-12 16:06:12,171 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,171 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:12,172 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:12,172 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,172 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:12,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:12,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:12,173 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,173 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:12,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:12,174 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testRestoreSnapshot type FLUSH 2023-07-12 16:06:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:12,174 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testRestoreSnapshot type FLUSH 2023-07-12 16:06:12,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:06:12,174 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:06:12,174 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:12,177 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:06:12,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:06:12,177 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:06:12,178 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:06:12,178 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:12,178 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:06:12,178 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:06:12,178 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:06:12,178 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:06:12,180 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,180 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:12,182 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:12,182 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,182 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:12,182 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,182 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:12,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:12,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName2' on coordinator 2023-07-12 16:06:12,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6febbe6f[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:12,184 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName2' starting 'in-barrier' execution. 2023-07-12 16:06:12,184 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,185 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,185 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,185 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:06:12,185 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:06:12,185 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,185 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:06:12,185 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:12,186 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:12,187 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:12,187 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:12,187 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:12,187 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. started... 2023-07-12 16:06:12,188 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:06:12,188 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:12,188 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:06:12,188 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:12,188 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.' region-info for snapshot=snapshotName2 2023-07-12 16:06:12,188 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:12,188 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8] hfiles 2023-07-12 16:06:12,188 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8 for snapshot=snapshotName2 2023-07-12 16:06:12,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:12,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:12,599 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. completed. 2023-07-12 16:06:12,599 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:12,600 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:12,600 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:12,600 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,600 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:06:12,600 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:12,602 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:06:12,602 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,602 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,602 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:06:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:12,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName2' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:06:12,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName2', counting down latch. Waiting for 0 more 2023-07-12 16:06:12,606 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName2' execution completed 2023-07-12 16:06:12,607 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:12,607 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:12,607 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName2 2023-07-12 16:06:12,607 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:12,610 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,610 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:12,610 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,610 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,610 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:12,611 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:12,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:12,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,611 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:12,613 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:12,614 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,618 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,618 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:12,618 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:12,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:12,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:12,618 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:12,619 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:12,618 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:12,618 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:12,619 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,619 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName2 2023-07-12 16:06:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:12,619 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName2 2023-07-12 16:06:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:12,619 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:12,619 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:12,620 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,620 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:12,620 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:12,620 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:12,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:12,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:12,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:12,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:12,621 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:12,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:12,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:13,045 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName2 2023-07-12 16:06:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:13,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:13,875 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName2 of table testRestoreSnapshot completed 2023-07-12 16:06:13,875 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:13,876 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:13,876 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot at 1689177971754Consolidate snapshot: snapshotName2 at 1689177972619 (+865 ms)Loading Region manifests for snapshotName2 at 1689177972619Writing data manifest for snapshotName2 at 1689177972624 (+5 ms)Verifying snapshot: snapshotName2 at 1689177973035 (+411 ms)Snapshot snapshotName2 of table testRestoreSnapshot completed at 1689177973875 (+840 ms) 2023-07-12 16:06:13,877 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=15, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:06:13,878 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED in 2.1220 sec 2023-07-12 16:06:15,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:15,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:15,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testRestoreSnapshot 2023-07-12 16:06:15,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:15,847 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177975847"}]},"ts":"1689177975847"} 2023-07-12 16:06:15,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=16 2023-07-12 16:06:15,852 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:06:15,854 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot to state=DISABLING 2023-07-12 16:06:15,855 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN}] 2023-07-12 16:06:15,857 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN 2023-07-12 16:06:15,858 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:15,858 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177975858"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177975858"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177975858"}]},"ts":"1689177975858"} 2023-07-12 16:06:15,859 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; CloseRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:15,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=16 2023-07-12 16:06:16,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:16,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 7def597ffc2b8cfcdf545c2c6ab26304, disabling compactions & flushes 2023-07-12 16:06:16,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:16,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:16,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. after waiting 0 ms 2023-07-12 16:06:16,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:16,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/recovered.edits/3008.seqid, newMaxSeqId=3008, maxSeqId=1 2023-07-12 16:06:16,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:16,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:16,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:16,032 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=CLOSED 2023-07-12 16:06:16,032 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177976032"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177976032"}]},"ts":"1689177976032"} 2023-07-12 16:06:16,035 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-12 16:06:16,035 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; CloseRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228 in 174 msec 2023-07-12 16:06:16,036 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-07-12 16:06:16,037 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN in 180 msec 2023-07-12 16:06:16,037 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177976037"}]},"ts":"1689177976037"} 2023-07-12 16:06:16,038 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:06:16,040 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot to state=DISABLED 2023-07-12 16:06:16,042 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot in 203 msec 2023-07-12 16:06:16,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=16 2023-07-12 16:06:16,170 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:06:16,188 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.ConnectionUtils(586): Start fetching meta region location from registry 2023-07-12 16:06:16,189 DEBUG [ReadOnlyZKClient-127.0.0.1:55421@0x6b24fef6] client.ConnectionUtils(594): The fetched meta region location is [region=hbase:meta,,1.1588230740, hostname=jenkins-hbase5.apache.org,38637,1689177958270, seqNum=-1] 2023-07-12 16:06:16,194 DEBUG [RPCClient-NioEventLoopGroup-6-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:16,197 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:40228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:16,201 INFO [RPCClient-NioEventLoopGroup-6-5] client.RawAsyncHBaseAdmin(1961): Taking restore-failsafe snapshot: hbase-failsafe-snapshotName1-1689177976201 2023-07-12 16:06:16,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689177976202 (current time:1689177976202). 2023-07-12 16:06:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x3c2651d0 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:16,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@444b3552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:16,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:16,211 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:40238, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:16,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c2651d0 to 127.0.0.1:55421 2023-07-12 16:06:16,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:16,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:16,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:16,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:16,217 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=19, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:16,218 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=19, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178576218 2023-07-12 16:06:16,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:16,218 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snapshotName1-1689177976201 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot 2023-07-12 16:06:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:16,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:16,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:16,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:16,631 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=DISABLED ttl=0 } 2023-07-12 16:06:16,633 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-07-12 16:06:16,633 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-07-12 16:06:16,637 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8] hfiles 2023-07-12 16:06:16,637 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8 2023-07-12 16:06:16,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:16,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:17,051 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=DISABLED ttl=0 } as finished. 2023-07-12 16:06:17,051 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snapshotName1-1689177976201 2023-07-12 16:06:17,053 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/hbase-failsafe-snapshotName1-1689177976201 2023-07-12 16:06:17,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:17,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:17,507 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/hbase-failsafe-snapshotName1-1689177976201 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/hbase-failsafe-snapshotName1-1689177976201 2023-07-12 16:06:18,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:18,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:18,361 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snapshotName1-1689177976201 of table testRestoreSnapshot completed 2023-07-12 16:06:18,362 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/hbase-failsafe-snapshotName1-1689177976201 2023-07-12 16:06:18,362 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/hbase-failsafe-snapshotName1-1689177976201 2023-07-12 16:06:18,362 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snapshotName1-1689177976201 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot at 1689177976218Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=DISABLED ttl=0 } at 1689177976631 (+413 ms)Consolidate snapshot: hbase-failsafe-snapshotName1-1689177976201 at 1689177977051 (+420 ms)Loading Region manifests for hbase-failsafe-snapshotName1-1689177976201 at 1689177977052 (+1 ms)Writing data manifest for hbase-failsafe-snapshotName1-1689177976201 at 1689177977056 (+4 ms)Verifying snapshot: hbase-failsafe-snapshotName1-1689177976201 at 1689177977498 (+442 ms)Snapshot hbase-failsafe-snapshotName1-1689177976201 of table testRestoreSnapshot completed at 1689177978361 (+863 ms) 2023-07-12 16:06:18,363 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=19, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:06:18,365 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE in 2.1470 sec 2023-07-12 16:06:20,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:20,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snapshotName1-1689177976201 table=testRestoreSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=20, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot snapshot=name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689177967641 type: FLUSH version: 2 ttl: 0 ) 2023-07-12 16:06:20,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(880): Restore snapshot=snapshotName1 as table=testRestoreSnapshot 2023-07-12 16:06:20,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=20 2023-07-12 16:06:20,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=20 2023-07-12 16:06:20,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=20 2023-07-12 16:06:20,739 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000002 2023-07-12 16:06:20,741 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:20,741 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000002 2023-07-12 16:06:20,742 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(384): Starting restore snapshot={ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:20,747 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689177967641 type: FLUSH version: 2 ttl: 0 2023-07-12 16:06:20,747 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot 2023-07-12 16:06:20,751 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 1 regions for table=testRestoreSnapshot 2023-07-12 16:06:20,751 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:20,754 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689177967641 type: FLUSH version: 2 ttl: 0 2023-07-12 16:06:20,755 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689177980755"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:20,797 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:06:20,797 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:20,797 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177980756"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177980756"}]},"ts":"1689177980756"} 2023-07-12 16:06:20,798 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:20,798 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 1 regions to Meta 2023-07-12 16:06:20,798 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:20,799 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(457): Restore snapshot={ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } on table=testRestoreSnapshot completed! 2023-07-12 16:06:20,801 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot snapshot=name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689177967641 type: FLUSH version: 2 ttl: 0 ) in 488 msec 2023-07-12 16:06:20,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=20 2023-07-12 16:06:20,950 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin(1985): Deleting restore-failsafe snapshot: hbase-failsafe-snapshotName1-1689177976201 2023-07-12 16:06:20,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "hbase-failsafe-snapshotName1-1689177976201" type: DISABLED 2023-07-12 16:06:20,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snapshotName1-1689177976201 2023-07-12 16:06:20,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$10(2385): Client=jenkins//172.31.10.131 enable testRestoreSnapshot 2023-07-12 16:06:20,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:20,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 16:06:20,976 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot 2023-07-12 16:06:20,977 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177980976"}]},"ts":"1689177980976"} 2023-07-12 16:06:20,978 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:06:20,981 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-12 16:06:20,981 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN}] 2023-07-12 16:06:20,983 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN 2023-07-12 16:06:20,983 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=22, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-07-12 16:06:21,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 16:06:21,134 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:21,134 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:21,134 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:21,134 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:21,135 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:21,135 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177981135"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177981135"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177981135"}]},"ts":"1689177981135"} 2023-07-12 16:06:21,137 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; OpenRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:21,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 16:06:21,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:21,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:21,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:21,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,295 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,296 DEBUG [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1 2023-07-12 16:06:21,296 DEBUG [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1 2023-07-12 16:06:21,297 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7def597ffc2b8cfcdf545c2c6ab26304 columnFamilyName f1 2023-07-12 16:06:21,306 DEBUG [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] regionserver.HStore(539): loaded hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8 2023-07-12 16:06:21,307 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] regionserver.HStore(310): Store=7def597ffc2b8cfcdf545c2c6ab26304/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:21,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 7def597ffc2b8cfcdf545c2c6ab26304; next sequenceid=3009; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75385193, jitterRate=0.12332691252231598}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:21,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:21,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304., pid=23, masterSystemTime=1689177981288 2023-07-12 16:06:21,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:21,318 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:21,318 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=OPEN, openSeqNum=3009, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:21,319 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177981318"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689177981318"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689177981318"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689177981318"}]},"ts":"1689177981318"} 2023-07-12 16:06:21,321 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 16:06:21,322 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; OpenRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228 in 183 msec 2023-07-12 16:06:21,323 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-07-12 16:06:21,325 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN in 340 msec 2023-07-12 16:06:21,326 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177981326"}]},"ts":"1689177981326"} 2023-07-12 16:06:21,327 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:06:21,329 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot' was successfully enabled. 2023-07-12 16:06:21,331 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot in 361 msec 2023-07-12 16:06:21,567 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:06:21,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 16:06:21,601 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: ENABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:06:21,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testRestoreSnapshot 2023-07-12 16:06:21,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:21,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=24 2023-07-12 16:06:21,681 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177981681"}]},"ts":"1689177981681"} 2023-07-12 16:06:21,683 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:06:21,685 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot to state=DISABLING 2023-07-12 16:06:21,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN}] 2023-07-12 16:06:21,687 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN 2023-07-12 16:06:21,687 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:21,688 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177981687"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177981687"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177981687"}]},"ts":"1689177981687"} 2023-07-12 16:06:21,689 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:21,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=24 2023-07-12 16:06:21,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 7def597ffc2b8cfcdf545c2c6ab26304, disabling compactions & flushes 2023-07-12 16:06:21,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:21,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:21,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. after waiting 0 ms 2023-07-12 16:06:21,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:21,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/recovered.edits/3011.seqid, newMaxSeqId=3011, maxSeqId=3008 2023-07-12 16:06:21,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:21,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:21,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:21,851 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=CLOSED 2023-07-12 16:06:21,851 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177981850"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177981850"}]},"ts":"1689177981850"} 2023-07-12 16:06:21,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-07-12 16:06:21,853 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228 in 163 msec 2023-07-12 16:06:21,855 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-12 16:06:21,855 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN in 168 msec 2023-07-12 16:06:21,856 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177981855"}]},"ts":"1689177981855"} 2023-07-12 16:06:21,856 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:06:21,858 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot to state=DISABLED 2023-07-12 16:06:21,860 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot in 182 msec 2023-07-12 16:06:21,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=24 2023-07-12 16:06:22,000 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:06:22,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot snapshot=name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689177971741 type: FLUSH version: 2 ttl: 0 ) 2023-07-12 16:06:22,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(880): Restore snapshot=snapshotName2 as table=testRestoreSnapshot 2023-07-12 16:06:22,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=27 2023-07-12 16:06:22,034 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000003 2023-07-12 16:06:22,035 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000002 2023-07-12 16:06:22,035 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000003 2023-07-12 16:06:22,036 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(384): Starting restore snapshot={ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:22,039 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689177971741 type: FLUSH version: 2 ttl: 0 2023-07-12 16:06:22,039 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot 2023-07-12 16:06:22,042 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 1 regions for table=testRestoreSnapshot 2023-07-12 16:06:22,042 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,044 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689177971741 type: FLUSH version: 2 ttl: 0 2023-07-12 16:06:22,045 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689177982045"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:22,047 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:06:22,047 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:22,047 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177982046"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177982046"}]},"ts":"1689177982046"} 2023-07-12 16:06:22,048 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:22,048 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 1 regions to Meta 2023-07-12 16:06:22,048 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:22,048 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(457): Restore snapshot={ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } on table=testRestoreSnapshot completed! 2023-07-12 16:06:22,050 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot snapshot=name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689177971741 type: FLUSH version: 2 ttl: 0 ) in 32 msec 2023-07-12 16:06:22,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=27 2023-07-12 16:06:22,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$10(2385): Client=jenkins//172.31.10.131 enable testRestoreSnapshot 2023-07-12 16:06:22,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:22,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=28 2023-07-12 16:06:22,134 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot 2023-07-12 16:06:22,134 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177982134"}]},"ts":"1689177982134"} 2023-07-12 16:06:22,135 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:06:22,138 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-12 16:06:22,138 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN}] 2023-07-12 16:06:22,139 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN 2023-07-12 16:06:22,140 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-07-12 16:06:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=28 2023-07-12 16:06:22,290 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:22,290 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:22,290 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:22,291 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:22,291 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:22,292 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177982291"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177982291"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177982291"}]},"ts":"1689177982291"} 2023-07-12 16:06:22,293 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:22,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:22,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:22,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=28 2023-07-12 16:06:22,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:22,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,451 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,452 DEBUG [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1 2023-07-12 16:06:22,452 DEBUG [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1 2023-07-12 16:06:22,453 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7def597ffc2b8cfcdf545c2c6ab26304 columnFamilyName f1 2023-07-12 16:06:22,460 DEBUG [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] regionserver.HStore(539): loaded hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8 2023-07-12 16:06:22,460 INFO [StoreOpener-7def597ffc2b8cfcdf545c2c6ab26304-1] regionserver.HStore(310): Store=7def597ffc2b8cfcdf545c2c6ab26304/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:22,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 7def597ffc2b8cfcdf545c2c6ab26304; next sequenceid=3012; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68179298, jitterRate=0.015950709581375122}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:22,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:22,469 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304., pid=30, masterSystemTime=1689177982445 2023-07-12 16:06:22,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:22,471 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:22,472 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=OPEN, openSeqNum=3012, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:22,472 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177982471"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689177982471"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689177982471"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689177982471"}]},"ts":"1689177982471"} 2023-07-12 16:06:22,475 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-07-12 16:06:22,475 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228 in 180 msec 2023-07-12 16:06:22,476 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-07-12 16:06:22,477 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, ASSIGN in 337 msec 2023-07-12 16:06:22,477 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177982477"}]},"ts":"1689177982477"} 2023-07-12 16:06:22,478 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:06:22,480 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot' was successfully enabled. 2023-07-12 16:06:22,482 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot in 350 msec 2023-07-12 16:06:22,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=28 2023-07-12 16:06:22,760 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: ENABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:06:22,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689177967641 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:06:22,790 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689177971741 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:06:22,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:06:22,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName2 2023-07-12 16:06:22,802 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testRestoreSnapshot 2023-07-12 16:06:22,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 16:06:22,805 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177982805"}]},"ts":"1689177982805"} 2023-07-12 16:06:22,806 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:06:22,809 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot to state=DISABLING 2023-07-12 16:06:22,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN}] 2023-07-12 16:06:22,811 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN 2023-07-12 16:06:22,812 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:22,812 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177982811"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177982811"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177982811"}]},"ts":"1689177982811"} 2023-07-12 16:06:22,813 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:22,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 16:06:22,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 7def597ffc2b8cfcdf545c2c6ab26304, disabling compactions & flushes 2023-07-12 16:06:22,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:22,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:22,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. after waiting 0 ms 2023-07-12 16:06:22,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:22,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/recovered.edits/3014.seqid, newMaxSeqId=3014, maxSeqId=3011 2023-07-12 16:06:22,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. 2023-07-12 16:06:22,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 7def597ffc2b8cfcdf545c2c6ab26304: 2023-07-12 16:06:22,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:22,976 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=7def597ffc2b8cfcdf545c2c6ab26304, regionState=CLOSED 2023-07-12 16:06:22,976 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177982976"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177982976"}]},"ts":"1689177982976"} 2023-07-12 16:06:22,979 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-07-12 16:06:22,979 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 7def597ffc2b8cfcdf545c2c6ab26304, server=jenkins-hbase5.apache.org,33647,1689177958228 in 164 msec 2023-07-12 16:06:22,981 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-07-12 16:06:22,981 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=7def597ffc2b8cfcdf545c2c6ab26304, UNASSIGN in 169 msec 2023-07-12 16:06:22,982 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177982981"}]},"ts":"1689177982981"} 2023-07-12 16:06:22,983 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:06:22,985 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot to state=DISABLED 2023-07-12 16:06:22,986 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot in 183 msec 2023-07-12 16:06:23,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 16:06:23,120 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:06:23,125 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testRestoreSnapshot 2023-07-12 16:06:23,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:23,135 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:23,136 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=34 2023-07-12 16:06:23,142 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:23,145 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/recovered.edits] 2023-07-12 16:06:23,154 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/f1/61528cc5de4648dba74da5eccc2292e8 2023-07-12 16:06:23,158 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/recovered.edits/3014.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304/recovered.edits/3014.seqid 2023-07-12 16:06:23,159 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/7def597ffc2b8cfcdf545c2c6ab26304 2023-07-12 16:06:23,159 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot regions 2023-07-12 16:06:23,162 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:23,185 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRestoreSnapshot from hbase:meta 2023-07-12 16:06:23,187 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot' descriptor. 2023-07-12 16:06:23,189 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:23,189 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot' from region states. 2023-07-12 16:06:23,189 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689177983189"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:23,191 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:06:23,191 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7def597ffc2b8cfcdf545c2c6ab26304, NAME => 'testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:23,191 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot' as deleted. 2023-07-12 16:06:23,191 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689177983191"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:23,193 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot state from META 2023-07-12 16:06:23,195 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:06:23,196 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot in 69 msec 2023-07-12 16:06:23,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=34 2023-07-12 16:06:23,240 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testRestoreSnapshot 2023-07-12 16:06:23,240 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:06:23,263 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testRestoreSnapshot[0] Thread=283 (was 265) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1649412395_17 at /127.0.0.1:32902 [Waiting for operation #34] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x168f361e-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-751465318_17 at /127.0.0.1:39564 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-751465318_17 at /127.0.0.1:39542 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async-Client-Retry-Timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/cluster_fd62e4bd-2d7f-c592-2cc1-3edac70c12cd/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x49280002-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool.commonPool-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/cluster_fd62e4bd-2d7f-c592-2cc1-3edac70c12cd/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x168f361e-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=544 (was 528) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=149 (was 166), ProcessCount=161 (was 161), AvailableMemoryMB=7320 (was 7367) 2023-07-12 16:06:23,274 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testTakeSnapshot[0] Thread=283, OpenFileDescriptor=544, MaxFileDescriptor=60000, SystemLoadAverage=149, ProcessCount=161, AvailableMemoryMB=7320 2023-07-12 16:06:23,277 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testTakeSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:06:23,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testTakeSnapshot 2023-07-12 16:06:23,279 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:06:23,279 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testTakeSnapshot" procId is: 35 2023-07-12 16:06:23,280 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:06:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-07-12 16:06:23,282 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:23,283 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb empty. 2023-07-12 16:06:23,283 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:23,283 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTakeSnapshot regions 2023-07-12 16:06:23,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-07-12 16:06:23,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-07-12 16:06:23,696 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:23,698 INFO [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => 91762ed77644238ae517369c8db1a1cb, NAME => 'testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testTakeSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:06:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-07-12 16:06:24,108 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(866): Instantiated testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:24,109 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1604): Closing 91762ed77644238ae517369c8db1a1cb, disabling compactions & flushes 2023-07-12 16:06:24,109 INFO [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1626): Closing region testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:24,109 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:24,109 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. after waiting 0 ms 2023-07-12 16:06:24,109 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:24,109 INFO [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1838): Closed testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:24,109 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for 91762ed77644238ae517369c8db1a1cb: 2023-07-12 16:06:24,112 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:06:24,113 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689177984113"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177984113"}]},"ts":"1689177984113"} 2023-07-12 16:06:24,114 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:24,115 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:06:24,115 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177984115"}]},"ts":"1689177984115"} 2023-07-12 16:06:24,116 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTakeSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:06:24,124 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:24,125 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:24,125 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:24,125 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:24,125 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTakeSnapshot, region=91762ed77644238ae517369c8db1a1cb, ASSIGN}] 2023-07-12 16:06:24,127 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTakeSnapshot, region=91762ed77644238ae517369c8db1a1cb, ASSIGN 2023-07-12 16:06:24,127 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTakeSnapshot, region=91762ed77644238ae517369c8db1a1cb, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,38637,1689177958270; forceNewPlan=false, retain=false 2023-07-12 16:06:24,278 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:06:24,279 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=91762ed77644238ae517369c8db1a1cb, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:24,279 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689177984279"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177984279"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177984279"}]},"ts":"1689177984279"} 2023-07-12 16:06:24,280 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 91762ed77644238ae517369c8db1a1cb, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:06:24,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-07-12 16:06:24,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:24,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 91762ed77644238ae517369c8db1a1cb, NAME => 'testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:24,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTakeSnapshot 91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:24,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:24,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:24,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:24,442 INFO [StoreOpener-91762ed77644238ae517369c8db1a1cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:24,449 DEBUG [StoreOpener-91762ed77644238ae517369c8db1a1cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1 2023-07-12 16:06:24,449 DEBUG [StoreOpener-91762ed77644238ae517369c8db1a1cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1 2023-07-12 16:06:24,450 INFO [StoreOpener-91762ed77644238ae517369c8db1a1cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 91762ed77644238ae517369c8db1a1cb columnFamilyName f1 2023-07-12 16:06:24,450 INFO [StoreOpener-91762ed77644238ae517369c8db1a1cb-1] regionserver.HStore(310): Store=91762ed77644238ae517369c8db1a1cb/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:24,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:24,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:24,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:24,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:06:24,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 91762ed77644238ae517369c8db1a1cb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62898123, jitterRate=-0.06274493038654327}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:24,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 91762ed77644238ae517369c8db1a1cb: 2023-07-12 16:06:24,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb., pid=37, masterSystemTime=1689177984432 2023-07-12 16:06:24,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:24,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:24,465 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=91762ed77644238ae517369c8db1a1cb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:24,465 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689177984465"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689177984465"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689177984465"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689177984465"}]},"ts":"1689177984465"} 2023-07-12 16:06:24,468 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-07-12 16:06:24,468 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 91762ed77644238ae517369c8db1a1cb, server=jenkins-hbase5.apache.org,38637,1689177958270 in 187 msec 2023-07-12 16:06:24,469 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-07-12 16:06:24,470 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testTakeSnapshot, region=91762ed77644238ae517369c8db1a1cb, ASSIGN in 343 msec 2023-07-12 16:06:24,470 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:06:24,471 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177984470"}]},"ts":"1689177984470"} 2023-07-12 16:06:24,472 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testTakeSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:06:24,474 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:06:24,475 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=testTakeSnapshot in 1.1970 sec 2023-07-12 16:06:25,078 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTakeSnapshot' 2023-07-12 16:06:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=35 2023-07-12 16:06:25,386 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testTakeSnapshot, procId: 35 completed 2023-07-12 16:06:25,386 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testTakeSnapshot get assigned. Timeout = 60000ms 2023-07-12 16:06:25,386 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:25,389 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testTakeSnapshot assigned to meta. Checking AM states. 2023-07-12 16:06:25,389 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:25,389 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testTakeSnapshot assigned. 2023-07-12 16:06:26,796 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:06:27,183 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:27,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689177987183 (current time:1689177987183). 2023-07-12 16:06:27,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:27,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x69787fd1 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@555535a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:27,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:27,192 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:53616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69787fd1 to 127.0.0.1:55421 2023-07-12 16:06:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:27,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:27,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:27,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:06:27,197 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=38, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:06:27,198 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=38, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178587198 2023-07-12 16:06:27,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:27,198 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testTakeSnapshot 2023-07-12 16:06:27,200 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=38, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:06:27,200 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=39, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:06:27,201 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE in 4 msec 2023-07-12 16:06:27,201 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=39, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:06:27,202 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=39, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED; timeout=600000, timestamp=1689178587201 2023-07-12 16:06:27,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:27,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:27,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:27,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:27,621 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName1 was in running list but was completed. Accepting new attempt. 2023-07-12 16:06:27,621 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:06:27,621 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:06:27,621 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:27,621 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:06:27,621 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:06:27,622 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,622 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:27,623 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,623 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:27,623 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:27,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:27,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:27,623 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:27,623 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:27,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 50 2023-07-12 16:06:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:27,625 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testTakeSnapshot type FLUSH 2023-07-12 16:06:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 50 2023-07-12 16:06:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:27,625 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testTakeSnapshot type FLUSH 2023-07-12 16:06:27,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:06:27,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:27,625 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:27,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:06:27,625 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:27,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:27,626 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:27,626 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:27,626 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:27,626 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:27,626 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:27,626 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:27,626 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:27,626 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:27,628 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,628 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:27,628 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,629 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,629 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:27,629 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,629 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:27,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:06:27,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c4c20c4[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:27,631 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:06:27,631 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,633 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,633 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,633 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:27,633 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,633 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,633 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:27,633 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:27,633 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:27,633 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:27,633 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:27,633 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:27,634 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. started... 2023-07-12 16:06:27,634 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2745): Flushing 91762ed77644238ae517369c8db1a1cb 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:06:27,641 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:27,641 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:27,641 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:27,653 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/.tmp/f1/9c7e48899bf647999a46ab2994c1cfcd 2023-07-12 16:06:27,662 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/.tmp/f1/9c7e48899bf647999a46ab2994c1cfcd as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1/9c7e48899bf647999a46ab2994c1cfcd 2023-07-12 16:06:27,669 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1/9c7e48899bf647999a46ab2994c1cfcd, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:06:27,671 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for 91762ed77644238ae517369c8db1a1cb in 37ms, sequenceid=3004, compaction requested=false 2023-07-12 16:06:27,671 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 91762ed77644238ae517369c8db1a1cb: 2023-07-12 16:06:27,672 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.' region-info for snapshot=snapshotName1 2023-07-12 16:06:27,672 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:27,672 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1/9c7e48899bf647999a46ab2994c1cfcd] hfiles 2023-07-12 16:06:27,672 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1/9c7e48899bf647999a46ab2994c1cfcd for snapshot=snapshotName1 2023-07-12 16:06:27,682 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. completed. 2023-07-12 16:06:27,683 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:27,683 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:27,683 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:27,683 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,683 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:27,683 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:27,685 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:27,685 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,685 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:27,685 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:06:27,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:06:27,689 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:06:27,690 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:27,690 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:27,690 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:06:27,690 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:27,691 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,692 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,692 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,692 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:27,692 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:27,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:27,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:27,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:27,693 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:27,694 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,696 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:27,696 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,701 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,701 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:27,701 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:27,702 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:27,702 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:27,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:27,702 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:27,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:27,702 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:06:27,702 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,702 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:06:27,702 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:27,702 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:27,703 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:27,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:27,703 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:27,703 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:27,703 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:27,703 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:27,703 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:27,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:27,723 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:06:27,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:27,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:28,549 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testTakeSnapshot completed 2023-07-12 16:06:28,550 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:28,550 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:28,550 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testTakeSnapshot at 1689177987198Consolidate snapshot: snapshotName1 at 1689177987702 (+504 ms)Loading Region manifests for snapshotName1 at 1689177987702Writing data manifest for snapshotName1 at 1689177987707 (+5 ms)Verifying snapshot: snapshotName1 at 1689177987715 (+8 ms)Snapshot snapshotName1 of table testTakeSnapshot completed at 1689177988549 (+834 ms) 2023-07-12 16:06:28,552 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=39, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:06:28,553 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED in 1.3530 sec 2023-07-12 16:06:29,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:29,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:29,251 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:29,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689177989251 (current time:1689177989251). 2023-07-12 16:06:29,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:29,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x22b69210 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@89ae760, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:29,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:29,271 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:53620, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22b69210 to 127.0.0.1:55421 2023-07-12 16:06:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:29,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:29,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:29,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=40, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:06:29,276 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=40, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:06:29,277 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=40, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178589277 2023-07-12 16:06:29,277 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testTakeSnapshot 2023-07-12 16:06:29,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:29,279 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=40, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:06:29,279 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=41, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:06:29,280 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE in 4 msec 2023-07-12 16:06:29,280 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=41, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:06:29,281 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=41, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED; timeout=600000, timestamp=1689178589281 2023-07-12 16:06:29,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:29,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:29,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:29,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:29,693 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName2 was in running list but was completed. Accepting new attempt. 2023-07-12 16:06:29,693 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName2 2023-07-12 16:06:29,693 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName2' 2023-07-12 16:06:29,693 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:29,693 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName2' starting 'acquire' 2023-07-12 16:06:29,693 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName2', kicking off acquire phase on members. 2023-07-12 16:06:29,694 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,694 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:29,695 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:29,695 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:29,695 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:29,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:29,696 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,696 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:29,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 50 2023-07-12 16:06:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:29,697 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testTakeSnapshot type FLUSH 2023-07-12 16:06:29,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:06:29,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:06:29,697 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:06:29,697 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:29,698 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:06:29,698 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:06:29,698 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:06:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:29,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,700 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 50 2023-07-12 16:06:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:29,700 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testTakeSnapshot type FLUSH 2023-07-12 16:06:29,700 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:06:29,700 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:29,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:06:29,700 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:06:29,700 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:29,701 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:06:29,701 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:06:29,701 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:06:29,704 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:29,704 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:29,705 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,705 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:29,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName2' on coordinator 2023-07-12 16:06:29,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71097b38[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:29,707 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName2' starting 'in-barrier' execution. 2023-07-12 16:06:29,707 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,708 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,709 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:06:29,709 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,709 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:29,709 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:29,709 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:29,709 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,709 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:29,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,709 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. started... 2023-07-12 16:06:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,710 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:06:29,710 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 91762ed77644238ae517369c8db1a1cb: 2023-07-12 16:06:29,710 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:06:29,710 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:29,710 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.' region-info for snapshot=snapshotName2 2023-07-12 16:06:29,710 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:29,710 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1/9c7e48899bf647999a46ab2994c1cfcd] hfiles 2023-07-12 16:06:29,710 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1/9c7e48899bf647999a46ab2994c1cfcd for snapshot=snapshotName2 2023-07-12 16:06:29,711 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:06:29,711 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:29,711 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:06:29,722 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. completed. 2023-07-12 16:06:29,722 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:29,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:29,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:29,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:06:29,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:29,724 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,724 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:06:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:29,724 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:29,725 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:06:29,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:29,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:29,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName2' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:06:29,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName2', counting down latch. Waiting for 0 more 2023-07-12 16:06:29,729 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName2' execution completed 2023-07-12 16:06:29,729 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:29,729 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:29,729 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName2 2023-07-12 16:06:29,729 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:29,732 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,732 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:29,732 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:29,732 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,733 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:29,734 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:29,734 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:29,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,736 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:29,737 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:29,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:29,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:29,740 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:29,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:29,740 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:29,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:29,741 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,740 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName2 2023-07-12 16:06:29,741 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:29,742 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName2 2023-07-12 16:06:29,742 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:29,742 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:29,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:29,743 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:29,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:29,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:30,166 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName2 2023-07-12 16:06:30,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:30,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:30,990 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName2 of table testTakeSnapshot completed 2023-07-12 16:06:30,990 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:30,990 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:30,990 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testTakeSnapshot at 1689177989277Consolidate snapshot: snapshotName2 at 1689177989742 (+465 ms)Loading Region manifests for snapshotName2 at 1689177989742Writing data manifest for snapshotName2 at 1689177989746 (+4 ms)Verifying snapshot: snapshotName2 at 1689177990157 (+411 ms)Snapshot snapshotName2 of table testTakeSnapshot completed at 1689177990990 (+833 ms) 2023-07-12 16:06:30,992 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=41, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:06:30,993 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED in 1.7140 sec 2023-07-12 16:06:31,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:31,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:31,349 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" table: "testTakeSnapshot" creation_time: 1689177987183 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:06:31,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName2" table: "testTakeSnapshot" creation_time: 1689177989251 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:06:31,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:06:31,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName2 2023-07-12 16:06:31,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testTakeSnapshot 2023-07-12 16:06:31,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=42, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testTakeSnapshot 2023-07-12 16:06:31,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=42 2023-07-12 16:06:31,359 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177991358"}]},"ts":"1689177991358"} 2023-07-12 16:06:31,360 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTakeSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:06:31,364 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testTakeSnapshot to state=DISABLING 2023-07-12 16:06:31,365 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTakeSnapshot, region=91762ed77644238ae517369c8db1a1cb, UNASSIGN}] 2023-07-12 16:06:31,366 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTakeSnapshot, region=91762ed77644238ae517369c8db1a1cb, UNASSIGN 2023-07-12 16:06:31,367 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=43 updating hbase:meta row=91762ed77644238ae517369c8db1a1cb, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:31,367 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689177991367"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177991367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177991367"}]},"ts":"1689177991367"} 2023-07-12 16:06:31,368 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE; CloseRegionProcedure 91762ed77644238ae517369c8db1a1cb, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:06:31,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=42 2023-07-12 16:06:31,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:31,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 91762ed77644238ae517369c8db1a1cb, disabling compactions & flushes 2023-07-12 16:06:31,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:31,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:31,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. after waiting 0 ms 2023-07-12 16:06:31,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:31,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/recovered.edits/3008.seqid, newMaxSeqId=3008, maxSeqId=1 2023-07-12 16:06:31,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. 2023-07-12 16:06:31,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 91762ed77644238ae517369c8db1a1cb: 2023-07-12 16:06:31,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:31,531 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=43 updating hbase:meta row=91762ed77644238ae517369c8db1a1cb, regionState=CLOSED 2023-07-12 16:06:31,531 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689177991531"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177991531"}]},"ts":"1689177991531"} 2023-07-12 16:06:31,533 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-07-12 16:06:31,534 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; CloseRegionProcedure 91762ed77644238ae517369c8db1a1cb, server=jenkins-hbase5.apache.org,38637,1689177958270 in 164 msec 2023-07-12 16:06:31,535 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-07-12 16:06:31,535 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; TransitRegionStateProcedure table=testTakeSnapshot, region=91762ed77644238ae517369c8db1a1cb, UNASSIGN in 169 msec 2023-07-12 16:06:31,536 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177991536"}]},"ts":"1689177991536"} 2023-07-12 16:06:31,540 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTakeSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:06:31,542 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testTakeSnapshot to state=DISABLED 2023-07-12 16:06:31,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, state=SUCCESS; DisableTableProcedure table=testTakeSnapshot in 189 msec 2023-07-12 16:06:31,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=42 2023-07-12 16:06:31,680 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testTakeSnapshot completed 2023-07-12 16:06:31,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testTakeSnapshot 2023-07-12 16:06:31,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=45, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:06:31,684 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=45, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:06:31,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=45 2023-07-12 16:06:31,684 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=45, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:06:31,691 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:31,694 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/recovered.edits] 2023-07-12 16:06:31,700 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1/9c7e48899bf647999a46ab2994c1cfcd to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/f1/9c7e48899bf647999a46ab2994c1cfcd 2023-07-12 16:06:31,705 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/recovered.edits/3008.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb/recovered.edits/3008.seqid 2023-07-12 16:06:31,706 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/91762ed77644238ae517369c8db1a1cb 2023-07-12 16:06:31,706 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testTakeSnapshot regions 2023-07-12 16:06:31,709 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=45, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:06:31,711 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testTakeSnapshot from hbase:meta 2023-07-12 16:06:31,718 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testTakeSnapshot' descriptor. 2023-07-12 16:06:31,719 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=45, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:06:31,720 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testTakeSnapshot' from region states. 2023-07-12 16:06:31,720 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689177991720"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:31,749 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:06:31,749 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 91762ed77644238ae517369c8db1a1cb, NAME => 'testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:31,749 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testTakeSnapshot' as deleted. 2023-07-12 16:06:31,749 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689177991749"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:31,751 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testTakeSnapshot state from META 2023-07-12 16:06:31,753 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=45, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:06:31,755 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, state=SUCCESS; DeleteTableProcedure table=testTakeSnapshot in 72 msec 2023-07-12 16:06:31,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=45 2023-07-12 16:06:31,790 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testTakeSnapshot 2023-07-12 16:06:31,790 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testTakeSnapshot completed 2023-07-12 16:06:31,807 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testTakeSnapshot[0] Thread=283 (was 283), OpenFileDescriptor=545 (was 544) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=169 (was 149) - SystemLoadAverage LEAK? -, ProcessCount=161 (was 161), AvailableMemoryMB=7211 (was 7320) 2023-07-12 16:06:31,820 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testCloneSnapshot[0] Thread=283, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=169, ProcessCount=161, AvailableMemoryMB=7211 2023-07-12 16:06:31,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testCloneSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:06:31,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCloneSnapshot 2023-07-12 16:06:31,825 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=46, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:06:31,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testCloneSnapshot" procId is: 46 2023-07-12 16:06:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=46 2023-07-12 16:06:31,827 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=46, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:06:31,829 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:31,830 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a empty. 2023-07-12 16:06:31,831 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:31,831 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testCloneSnapshot regions 2023-07-12 16:06:31,843 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:31,844 INFO [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4c2fefd5ec41ff99dc50f537aa39571a, NAME => 'testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCloneSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:06:31,853 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(866): Instantiated testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:31,853 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1604): Closing 4c2fefd5ec41ff99dc50f537aa39571a, disabling compactions & flushes 2023-07-12 16:06:31,853 INFO [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1626): Closing region testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:31,853 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:31,853 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. after waiting 0 ms 2023-07-12 16:06:31,853 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:31,853 INFO [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1838): Closed testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:31,853 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for 4c2fefd5ec41ff99dc50f537aa39571a: 2023-07-12 16:06:31,856 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=46, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:06:31,858 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689177991857"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177991857"}]},"ts":"1689177991857"} 2023-07-12 16:06:31,859 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:31,860 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=46, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:06:31,860 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177991860"}]},"ts":"1689177991860"} 2023-07-12 16:06:31,861 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:06:31,864 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:31,865 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:31,865 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:31,865 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:31,866 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=46, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCloneSnapshot, region=4c2fefd5ec41ff99dc50f537aa39571a, ASSIGN}] 2023-07-12 16:06:31,873 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, ppid=46, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCloneSnapshot, region=4c2fefd5ec41ff99dc50f537aa39571a, ASSIGN 2023-07-12 16:06:31,875 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=47, ppid=46, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCloneSnapshot, region=4c2fefd5ec41ff99dc50f537aa39571a, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,33647,1689177958228; forceNewPlan=false, retain=false 2023-07-12 16:06:31,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=46 2023-07-12 16:06:32,025 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:06:32,026 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=4c2fefd5ec41ff99dc50f537aa39571a, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:32,026 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689177992026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177992026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177992026"}]},"ts":"1689177992026"} 2023-07-12 16:06:32,028 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 4c2fefd5ec41ff99dc50f537aa39571a, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:32,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=46 2023-07-12 16:06:32,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:32,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4c2fefd5ec41ff99dc50f537aa39571a, NAME => 'testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:32,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCloneSnapshot 4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:32,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:32,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:32,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:32,186 INFO [StoreOpener-4c2fefd5ec41ff99dc50f537aa39571a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:32,188 DEBUG [StoreOpener-4c2fefd5ec41ff99dc50f537aa39571a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1 2023-07-12 16:06:32,188 DEBUG [StoreOpener-4c2fefd5ec41ff99dc50f537aa39571a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1 2023-07-12 16:06:32,189 INFO [StoreOpener-4c2fefd5ec41ff99dc50f537aa39571a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c2fefd5ec41ff99dc50f537aa39571a columnFamilyName f1 2023-07-12 16:06:32,190 INFO [StoreOpener-4c2fefd5ec41ff99dc50f537aa39571a-1] regionserver.HStore(310): Store=4c2fefd5ec41ff99dc50f537aa39571a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:32,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:32,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:32,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:32,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:06:32,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 4c2fefd5ec41ff99dc50f537aa39571a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61389957, jitterRate=-0.08521835505962372}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:32,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 4c2fefd5ec41ff99dc50f537aa39571a: 2023-07-12 16:06:32,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a., pid=48, masterSystemTime=1689177992180 2023-07-12 16:06:32,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:32,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:32,203 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=4c2fefd5ec41ff99dc50f537aa39571a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:32,203 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689177992203"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689177992203"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689177992203"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689177992203"}]},"ts":"1689177992203"} 2023-07-12 16:06:32,206 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-07-12 16:06:32,206 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; OpenRegionProcedure 4c2fefd5ec41ff99dc50f537aa39571a, server=jenkins-hbase5.apache.org,33647,1689177958228 in 176 msec 2023-07-12 16:06:32,207 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=46 2023-07-12 16:06:32,208 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=46, state=SUCCESS; TransitRegionStateProcedure table=testCloneSnapshot, region=4c2fefd5ec41ff99dc50f537aa39571a, ASSIGN in 340 msec 2023-07-12 16:06:32,209 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=46, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:06:32,209 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177992209"}]},"ts":"1689177992209"} 2023-07-12 16:06:32,210 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:06:32,212 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=46, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:06:32,214 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; CreateTableProcedure table=testCloneSnapshot in 389 msec 2023-07-12 16:06:32,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=46 2023-07-12 16:06:32,430 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCloneSnapshot, procId: 46 completed 2023-07-12 16:06:32,430 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testCloneSnapshot get assigned. Timeout = 60000ms 2023-07-12 16:06:32,431 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:32,433 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testCloneSnapshot assigned to meta. Checking AM states. 2023-07-12 16:06:32,433 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:32,433 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testCloneSnapshot assigned. 2023-07-12 16:06:34,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:34,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689177994314 (current time:1689177994314). 2023-07-12 16:06:34,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:34,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x2a7d2515 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:34,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d8464bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:34,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:34,323 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:53636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a7d2515 to 127.0.0.1:55421 2023-07-12 16:06:34,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:34,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:34,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:34,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE 2023-07-12 16:06:34,329 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=49, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE 2023-07-12 16:06:34,329 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=49, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178594329 2023-07-12 16:06:34,329 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testCloneSnapshot 2023-07-12 16:06:34,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } 2023-07-12 16:06:34,331 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED 2023-07-12 16:06:34,331 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=49, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE 2023-07-12 16:06:34,332 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE in 4 msec 2023-07-12 16:06:34,333 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=50, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED 2023-07-12 16:06:34,333 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=50, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED; timeout=600000, timestamp=1689178594333 2023-07-12 16:06:34,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:34,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:34,349 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName1 was in running list but was completed. Accepting new attempt. 2023-07-12 16:06:34,349 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:06:34,349 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:06:34,349 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:34,349 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:06:34,349 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:06:34,350 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,350 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:34,352 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:34,352 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:34,352 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:34,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:34,353 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,353 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:34,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:06:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:06:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:34,355 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testCloneSnapshot type FLUSH 2023-07-12 16:06:34,355 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testCloneSnapshot type FLUSH 2023-07-12 16:06:34,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:06:34,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:34,356 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:34,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:06:34,356 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:34,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:34,356 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:34,356 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:34,357 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:34,356 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:34,357 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:34,357 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:34,357 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:34,357 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:34,359 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,360 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,360 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:34,360 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,360 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:34,361 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:34,361 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:34,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:06:34,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f727331[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:34,363 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:06:34,364 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,365 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,365 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,365 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:34,366 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:34,366 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:34,366 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:34,366 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,366 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:34,366 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:34,366 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:34,366 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:34,366 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. started... 2023-07-12 16:06:34,367 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2745): Flushing 4c2fefd5ec41ff99dc50f537aa39571a 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:06:34,369 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:34,369 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:34,369 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:34,386 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/.tmp/f1/ff36d8d536ac4289a500a4aa7e0fa944 2023-07-12 16:06:34,403 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/.tmp/f1/ff36d8d536ac4289a500a4aa7e0fa944 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1/ff36d8d536ac4289a500a4aa7e0fa944 2023-07-12 16:06:34,415 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1/ff36d8d536ac4289a500a4aa7e0fa944, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:06:34,416 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for 4c2fefd5ec41ff99dc50f537aa39571a in 49ms, sequenceid=3004, compaction requested=false 2023-07-12 16:06:34,416 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCloneSnapshot' 2023-07-12 16:06:34,417 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 4c2fefd5ec41ff99dc50f537aa39571a: 2023-07-12 16:06:34,417 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.' region-info for snapshot=snapshotName1 2023-07-12 16:06:34,417 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:34,417 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1/ff36d8d536ac4289a500a4aa7e0fa944] hfiles 2023-07-12 16:06:34,417 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1/ff36d8d536ac4289a500a4aa7e0fa944 for snapshot=snapshotName1 2023-07-12 16:06:34,432 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. completed. 2023-07-12 16:06:34,432 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:34,432 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:34,433 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:34,433 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,433 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:34,433 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:34,435 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,435 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:34,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:34,435 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:34,435 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:06:34,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:06:34,440 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:06:34,440 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:34,440 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:34,440 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:06:34,440 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:34,443 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,443 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,443 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,443 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:34,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,443 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:34,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,444 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:34,445 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:34,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:34,448 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,449 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,453 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,453 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:34,453 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:34,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:34,453 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:34,453 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:34,453 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:34,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:34,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:34,453 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,454 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:34,454 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:34,454 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:34,454 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,454 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:06:34,454 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,455 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:06:34,455 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:34,455 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:34,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:34,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:34,456 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:34,481 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:06:34,502 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testCloneSnapshot completed 2023-07-12 16:06:34,502 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:34,502 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:34,503 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testCloneSnapshot at 1689177994330Consolidate snapshot: snapshotName1 at 1689177994455 (+125 ms)Loading Region manifests for snapshotName1 at 1689177994455Writing data manifest for snapshotName1 at 1689177994462 (+7 ms)Verifying snapshot: snapshotName1 at 1689177994469 (+7 ms)Snapshot snapshotName1 of table testCloneSnapshot completed at 1689177994502 (+33 ms) 2023-07-12 16:06:34,504 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=50, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED 2023-07-12 16:06:34,505 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED in 174 msec 2023-07-12 16:06:34,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:06:34,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:34,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=51, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testCloneSnapshot2 snapshot=name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689177994314 type: FLUSH version: 2 ttl: 0 ) 2023-07-12 16:06:34,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(750): Clone snapshot=snapshotName1 as table=testCloneSnapshot2 2023-07-12 16:06:34,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 16:06:34,612 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:34,616 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689177994314 type: FLUSH version: 2 ttl: 0 2023-07-12 16:06:34,616 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2 2023-07-12 16:06:34,622 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: 4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:34,623 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=4c2fefd5ec41ff99dc50f537aa39571a as 638537a34407bb4fa04b0e4031cc6f39 in snapshot snapshotName1 2023-07-12 16:06:34,624 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => 638537a34407bb4fa04b0e4031cc6f39, NAME => 'testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCloneSnapshot2', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:06:34,638 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(866): Instantiated testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:34,638 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink ff36d8d536ac4289a500a4aa7e0fa944 from cloned region in snapshot snapshotName1 to table=testCloneSnapshot2 2023-07-12 16:06:34,650 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1604): Closing 638537a34407bb4fa04b0e4031cc6f39, disabling compactions & flushes 2023-07-12 16:06:34,650 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1626): Closing region testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:34,650 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:34,650 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. after waiting 0 ms 2023-07-12 16:06:34,650 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:34,650 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1838): Closed testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:34,650 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for 638537a34407bb4fa04b0e4031cc6f39: 2023-07-12 16:06:34,651 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689177994314 type: FLUSH version: 2 ttl: 0 2023-07-12 16:06:34,651 INFO [PEWorker-5] procedure.CloneSnapshotProcedure$1(421): Clone snapshot=snapshotName1 on table=testCloneSnapshot2 completed! 2023-07-12 16:06:34,656 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689177994655"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177994655"}]},"ts":"1689177994655"} 2023-07-12 16:06:34,657 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:34,658 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177994658"}]},"ts":"1689177994658"} 2023-07-12 16:06:34,659 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot2, state=ENABLING in hbase:meta 2023-07-12 16:06:34,665 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:34,669 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:34,669 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:34,670 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:34,670 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCloneSnapshot2, region=638537a34407bb4fa04b0e4031cc6f39, ASSIGN}] 2023-07-12 16:06:34,672 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCloneSnapshot2, region=638537a34407bb4fa04b0e4031cc6f39, ASSIGN 2023-07-12 16:06:34,673 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCloneSnapshot2, region=638537a34407bb4fa04b0e4031cc6f39, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,38637,1689177958270; forceNewPlan=false, retain=false 2023-07-12 16:06:34,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 16:06:34,823 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:06:34,824 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=638537a34407bb4fa04b0e4031cc6f39, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:34,824 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689177994824"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177994824"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177994824"}]},"ts":"1689177994824"} 2023-07-12 16:06:34,825 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=53, ppid=52, state=RUNNABLE; OpenRegionProcedure 638537a34407bb4fa04b0e4031cc6f39, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:06:34,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 16:06:34,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:34,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 638537a34407bb4fa04b0e4031cc6f39, NAME => 'testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:34,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCloneSnapshot2 638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:34,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:34,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:34,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:34,983 INFO [StoreOpener-638537a34407bb4fa04b0e4031cc6f39-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:34,985 DEBUG [StoreOpener-638537a34407bb4fa04b0e4031cc6f39-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/f1 2023-07-12 16:06:34,985 DEBUG [StoreOpener-638537a34407bb4fa04b0e4031cc6f39-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/f1 2023-07-12 16:06:34,985 INFO [StoreOpener-638537a34407bb4fa04b0e4031cc6f39-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 638537a34407bb4fa04b0e4031cc6f39 columnFamilyName f1 2023-07-12 16:06:34,996 DEBUG [StoreOpener-638537a34407bb4fa04b0e4031cc6f39-1] regionserver.HStore(539): loaded hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/f1/testCloneSnapshot=4c2fefd5ec41ff99dc50f537aa39571a-ff36d8d536ac4289a500a4aa7e0fa944 2023-07-12 16:06:34,996 INFO [StoreOpener-638537a34407bb4fa04b0e4031cc6f39-1] regionserver.HStore(310): Store=638537a34407bb4fa04b0e4031cc6f39/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:34,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:34,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:35,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:35,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/recovered.edits/3004.seqid, newMaxSeqId=3004, maxSeqId=-1 2023-07-12 16:06:35,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 638537a34407bb4fa04b0e4031cc6f39; next sequenceid=3005; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74493867, jitterRate=0.11004512012004852}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:35,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 638537a34407bb4fa04b0e4031cc6f39: 2023-07-12 16:06:35,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39., pid=53, masterSystemTime=1689177994977 2023-07-12 16:06:35,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:35,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:35,011 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=638537a34407bb4fa04b0e4031cc6f39, regionState=OPEN, openSeqNum=3005, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:35,011 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689177995011"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689177995011"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689177995011"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689177995011"}]},"ts":"1689177995011"} 2023-07-12 16:06:35,013 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=53, resume processing ppid=52 2023-07-12 16:06:35,013 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=52, state=SUCCESS; OpenRegionProcedure 638537a34407bb4fa04b0e4031cc6f39, server=jenkins-hbase5.apache.org,38637,1689177958270 in 187 msec 2023-07-12 16:06:35,015 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-07-12 16:06:35,016 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; TransitRegionStateProcedure table=testCloneSnapshot2, region=638537a34407bb4fa04b0e4031cc6f39, ASSIGN in 343 msec 2023-07-12 16:06:35,016 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177995016"}]},"ts":"1689177995016"} 2023-07-12 16:06:35,017 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot2, state=ENABLED in hbase:meta 2023-07-12 16:06:35,021 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, state=SUCCESS; CloneSnapshotProcedure (table=testCloneSnapshot2 snapshot=name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689177994314 type: FLUSH version: 2 ttl: 0 ) in 440 msec 2023-07-12 16:06:35,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 16:06:35,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689177994314 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:06:35,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:06:35,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testCloneSnapshot 2023-07-12 16:06:35,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCloneSnapshot 2023-07-12 16:06:35,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=54 2023-07-12 16:06:35,234 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177995234"}]},"ts":"1689177995234"} 2023-07-12 16:06:35,235 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:06:35,237 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCloneSnapshot to state=DISABLING 2023-07-12 16:06:35,237 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCloneSnapshot, region=4c2fefd5ec41ff99dc50f537aa39571a, UNASSIGN}] 2023-07-12 16:06:35,239 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCloneSnapshot, region=4c2fefd5ec41ff99dc50f537aa39571a, UNASSIGN 2023-07-12 16:06:35,239 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4c2fefd5ec41ff99dc50f537aa39571a, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:35,239 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689177995239"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177995239"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177995239"}]},"ts":"1689177995239"} 2023-07-12 16:06:35,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; CloseRegionProcedure 4c2fefd5ec41ff99dc50f537aa39571a, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:06:35,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=54 2023-07-12 16:06:35,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:35,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 4c2fefd5ec41ff99dc50f537aa39571a, disabling compactions & flushes 2023-07-12 16:06:35,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:35,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:35,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. after waiting 0 ms 2023-07-12 16:06:35,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:35,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/recovered.edits/3007.seqid, newMaxSeqId=3007, maxSeqId=1 2023-07-12 16:06:35,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. 2023-07-12 16:06:35,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 4c2fefd5ec41ff99dc50f537aa39571a: 2023-07-12 16:06:35,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:35,401 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4c2fefd5ec41ff99dc50f537aa39571a, regionState=CLOSED 2023-07-12 16:06:35,402 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689177995401"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177995401"}]},"ts":"1689177995401"} 2023-07-12 16:06:35,404 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-07-12 16:06:35,404 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; CloseRegionProcedure 4c2fefd5ec41ff99dc50f537aa39571a, server=jenkins-hbase5.apache.org,33647,1689177958228 in 162 msec 2023-07-12 16:06:35,405 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-07-12 16:06:35,405 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=testCloneSnapshot, region=4c2fefd5ec41ff99dc50f537aa39571a, UNASSIGN in 167 msec 2023-07-12 16:06:35,406 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177995406"}]},"ts":"1689177995406"} 2023-07-12 16:06:35,406 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:06:35,409 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCloneSnapshot to state=DISABLED 2023-07-12 16:06:35,410 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; DisableTableProcedure table=testCloneSnapshot in 178 msec 2023-07-12 16:06:35,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=54 2023-07-12 16:06:35,550 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testCloneSnapshot completed 2023-07-12 16:06:35,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testCloneSnapshot2 2023-07-12 16:06:35,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCloneSnapshot2 2023-07-12 16:06:35,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=57 2023-07-12 16:06:35,554 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177995553"}]},"ts":"1689177995553"} 2023-07-12 16:06:35,554 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot2, state=DISABLING in hbase:meta 2023-07-12 16:06:35,556 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCloneSnapshot2 to state=DISABLING 2023-07-12 16:06:35,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCloneSnapshot2, region=638537a34407bb4fa04b0e4031cc6f39, UNASSIGN}] 2023-07-12 16:06:35,558 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=58, ppid=57, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCloneSnapshot2, region=638537a34407bb4fa04b0e4031cc6f39, UNASSIGN 2023-07-12 16:06:35,559 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=58 updating hbase:meta row=638537a34407bb4fa04b0e4031cc6f39, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:35,559 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689177995559"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177995559"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177995559"}]},"ts":"1689177995559"} 2023-07-12 16:06:35,560 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE; CloseRegionProcedure 638537a34407bb4fa04b0e4031cc6f39, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:06:35,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=57 2023-07-12 16:06:35,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:35,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 638537a34407bb4fa04b0e4031cc6f39, disabling compactions & flushes 2023-07-12 16:06:35,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:35,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:35,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. after waiting 0 ms 2023-07-12 16:06:35,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:35,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/recovered.edits/3007.seqid, newMaxSeqId=3007, maxSeqId=3004 2023-07-12 16:06:35,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39. 2023-07-12 16:06:35,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 638537a34407bb4fa04b0e4031cc6f39: 2023-07-12 16:06:35,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:35,720 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=58 updating hbase:meta row=638537a34407bb4fa04b0e4031cc6f39, regionState=CLOSED 2023-07-12 16:06:35,720 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689177995720"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177995720"}]},"ts":"1689177995720"} 2023-07-12 16:06:35,722 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-07-12 16:06:35,723 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; CloseRegionProcedure 638537a34407bb4fa04b0e4031cc6f39, server=jenkins-hbase5.apache.org,38637,1689177958270 in 161 msec 2023-07-12 16:06:35,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-07-12 16:06:35,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; TransitRegionStateProcedure table=testCloneSnapshot2, region=638537a34407bb4fa04b0e4031cc6f39, UNASSIGN in 166 msec 2023-07-12 16:06:35,724 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177995724"}]},"ts":"1689177995724"} 2023-07-12 16:06:35,725 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot2, state=DISABLED in hbase:meta 2023-07-12 16:06:35,727 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCloneSnapshot2 to state=DISABLED 2023-07-12 16:06:35,728 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; DisableTableProcedure table=testCloneSnapshot2 in 176 msec 2023-07-12 16:06:35,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=57 2023-07-12 16:06:35,870 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testCloneSnapshot2 completed 2023-07-12 16:06:35,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testCloneSnapshot 2023-07-12 16:06:35,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=60, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:06:35,873 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=60, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:06:35,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=60 2023-07-12 16:06:35,873 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=60, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:06:35,878 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:35,881 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/recovered.edits] 2023-07-12 16:06:35,885 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1/ff36d8d536ac4289a500a4aa7e0fa944 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/f1/ff36d8d536ac4289a500a4aa7e0fa944 2023-07-12 16:06:35,889 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/recovered.edits/3007.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a/recovered.edits/3007.seqid 2023-07-12 16:06:35,889 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/4c2fefd5ec41ff99dc50f537aa39571a 2023-07-12 16:06:35,890 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testCloneSnapshot regions 2023-07-12 16:06:35,892 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=60, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:06:35,894 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testCloneSnapshot from hbase:meta 2023-07-12 16:06:35,896 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testCloneSnapshot' descriptor. 2023-07-12 16:06:35,897 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=60, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:06:35,897 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testCloneSnapshot' from region states. 2023-07-12 16:06:35,897 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689177995897"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:35,899 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:06:35,899 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4c2fefd5ec41ff99dc50f537aa39571a, NAME => 'testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:35,899 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testCloneSnapshot' as deleted. 2023-07-12 16:06:35,899 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689177995899"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:35,900 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testCloneSnapshot state from META 2023-07-12 16:06:35,902 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=60, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:06:35,903 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=60, state=SUCCESS; DeleteTableProcedure table=testCloneSnapshot in 31 msec 2023-07-12 16:06:35,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=60 2023-07-12 16:06:35,979 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testCloneSnapshot 2023-07-12 16:06:35,980 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testCloneSnapshot completed 2023-07-12 16:06:35,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testCloneSnapshot2 2023-07-12 16:06:35,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=61, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:06:35,982 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=61, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:06:35,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=61 2023-07-12 16:06:35,983 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=61, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:06:35,987 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:35,989 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/recovered.edits] 2023-07-12 16:06:35,995 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/f1/testCloneSnapshot=4c2fefd5ec41ff99dc50f537aa39571a-ff36d8d536ac4289a500a4aa7e0fa944 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/f1/testCloneSnapshot=4c2fefd5ec41ff99dc50f537aa39571a-ff36d8d536ac4289a500a4aa7e0fa944 2023-07-12 16:06:35,999 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/recovered.edits/3007.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39/recovered.edits/3007.seqid 2023-07-12 16:06:36,000 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/638537a34407bb4fa04b0e4031cc6f39 2023-07-12 16:06:36,000 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCloneSnapshot2 regions 2023-07-12 16:06:36,004 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=61, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:06:36,005 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testCloneSnapshot2 from hbase:meta 2023-07-12 16:06:36,006 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testCloneSnapshot2' descriptor. 2023-07-12 16:06:36,007 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=61, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:06:36,007 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testCloneSnapshot2' from region states. 2023-07-12 16:06:36,008 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689177996008"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:36,009 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:06:36,009 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 638537a34407bb4fa04b0e4031cc6f39, NAME => 'testCloneSnapshot2,,1689177991823.638537a34407bb4fa04b0e4031cc6f39.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:36,009 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testCloneSnapshot2' as deleted. 2023-07-12 16:06:36,009 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689177996009"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:36,011 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testCloneSnapshot2 state from META 2023-07-12 16:06:36,014 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=61, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:06:36,015 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, state=SUCCESS; DeleteTableProcedure table=testCloneSnapshot2 in 35 msec 2023-07-12 16:06:36,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=61 2023-07-12 16:06:36,090 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testCloneSnapshot2 2023-07-12 16:06:36,090 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testCloneSnapshot2 completed 2023-07-12 16:06:36,104 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testCloneSnapshot[0] Thread=286 (was 283) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1649412395_17 at /127.0.0.1:48180 [Waiting for operation #21] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-751465318_17 at /127.0.0.1:48334 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_576297013_17 at /127.0.0.1:48298 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x49280002-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=543 (was 545), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=195 (was 169) - SystemLoadAverage LEAK? -, ProcessCount=161 (was 161), AvailableMemoryMB=7054 (was 7211) 2023-07-12 16:06:36,117 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testDeleteSnapshots[0] Thread=286, OpenFileDescriptor=543, MaxFileDescriptor=60000, SystemLoadAverage=195, ProcessCount=161, AvailableMemoryMB=7053 2023-07-12 16:06:36,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testDeleteSnapshots', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:06:36,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testDeleteSnapshots 2023-07-12 16:06:36,122 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=62, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:06:36,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testDeleteSnapshots" procId is: 62 2023-07-12 16:06:36,123 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=62, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:06:36,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=62 2023-07-12 16:06:36,124 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:36,125 DEBUG [HFileArchiver-9] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e empty. 2023-07-12 16:06:36,126 DEBUG [HFileArchiver-9] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:36,126 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testDeleteSnapshots regions 2023-07-12 16:06:36,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=62 2023-07-12 16:06:36,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=62 2023-07-12 16:06:36,538 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:36,539 INFO [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8f86f373a8ffe82c020499e3ca1d249e, NAME => 'testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testDeleteSnapshots', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:06:36,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=62 2023-07-12 16:06:36,946 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(866): Instantiated testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:36,947 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1604): Closing 8f86f373a8ffe82c020499e3ca1d249e, disabling compactions & flushes 2023-07-12 16:06:36,947 INFO [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1626): Closing region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:36,947 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:36,947 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1714): Acquired close lock on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. after waiting 0 ms 2023-07-12 16:06:36,947 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1724): Updates disabled for region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:36,947 INFO [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1838): Closed testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:36,947 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1558): Region close journal for 8f86f373a8ffe82c020499e3ca1d249e: 2023-07-12 16:06:36,949 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=62, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:06:36,950 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177996950"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689177996950"}]},"ts":"1689177996950"} 2023-07-12 16:06:36,951 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:36,952 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=62, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:06:36,952 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177996952"}]},"ts":"1689177996952"} 2023-07-12 16:06:36,953 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteSnapshots, state=ENABLING in hbase:meta 2023-07-12 16:06:36,956 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:36,957 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:36,957 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:36,957 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:36,957 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteSnapshots, region=8f86f373a8ffe82c020499e3ca1d249e, ASSIGN}] 2023-07-12 16:06:36,959 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteSnapshots, region=8f86f373a8ffe82c020499e3ca1d249e, ASSIGN 2023-07-12 16:06:36,959 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDeleteSnapshots, region=8f86f373a8ffe82c020499e3ca1d249e, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,38637,1689177958270; forceNewPlan=false, retain=false 2023-07-12 16:06:37,030 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:06:37,109 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:06:37,110 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=8f86f373a8ffe82c020499e3ca1d249e, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:37,111 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177997110"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689177997110"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689177997110"}]},"ts":"1689177997110"} 2023-07-12 16:06:37,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; OpenRegionProcedure 8f86f373a8ffe82c020499e3ca1d249e, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:06:37,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=62 2023-07-12 16:06:37,267 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:37,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8f86f373a8ffe82c020499e3ca1d249e, NAME => 'testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:37,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDeleteSnapshots 8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:37,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:37,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:37,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:37,269 INFO [StoreOpener-8f86f373a8ffe82c020499e3ca1d249e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:37,271 DEBUG [StoreOpener-8f86f373a8ffe82c020499e3ca1d249e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1 2023-07-12 16:06:37,271 DEBUG [StoreOpener-8f86f373a8ffe82c020499e3ca1d249e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1 2023-07-12 16:06:37,271 INFO [StoreOpener-8f86f373a8ffe82c020499e3ca1d249e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8f86f373a8ffe82c020499e3ca1d249e columnFamilyName f1 2023-07-12 16:06:37,272 INFO [StoreOpener-8f86f373a8ffe82c020499e3ca1d249e-1] regionserver.HStore(310): Store=8f86f373a8ffe82c020499e3ca1d249e/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:37,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:37,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:37,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:37,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:06:37,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 8f86f373a8ffe82c020499e3ca1d249e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61779917, jitterRate=-0.07940749824047089}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:37,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 8f86f373a8ffe82c020499e3ca1d249e: 2023-07-12 16:06:37,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e., pid=64, masterSystemTime=1689177997263 2023-07-12 16:06:37,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:37,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:37,282 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=8f86f373a8ffe82c020499e3ca1d249e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:37,282 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689177997282"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689177997282"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689177997282"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689177997282"}]},"ts":"1689177997282"} 2023-07-12 16:06:37,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-07-12 16:06:37,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; OpenRegionProcedure 8f86f373a8ffe82c020499e3ca1d249e, server=jenkins-hbase5.apache.org,38637,1689177958270 in 172 msec 2023-07-12 16:06:37,286 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-07-12 16:06:37,286 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=testDeleteSnapshots, region=8f86f373a8ffe82c020499e3ca1d249e, ASSIGN in 328 msec 2023-07-12 16:06:37,287 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=62, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:06:37,287 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689177997287"}]},"ts":"1689177997287"} 2023-07-12 16:06:37,288 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteSnapshots, state=ENABLED in hbase:meta 2023-07-12 16:06:37,290 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=62, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:06:37,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; CreateTableProcedure table=testDeleteSnapshots in 1.1700 sec 2023-07-12 16:06:38,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=62 2023-07-12 16:06:38,228 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testDeleteSnapshots, procId: 62 completed 2023-07-12 16:06:38,228 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testDeleteSnapshots get assigned. Timeout = 60000ms 2023-07-12 16:06:38,228 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:38,231 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testDeleteSnapshots assigned to meta. Checking AM states. 2023-07-12 16:06:38,231 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:38,231 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testDeleteSnapshots assigned. 2023-07-12 16:06:39,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:39,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689177999499 (current time:1689177999499). 2023-07-12 16:06:39,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:39,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x5bb78230 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:39,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@264ba061, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:39,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:39,509 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5bb78230 to 127.0.0.1:55421 2023-07-12 16:06:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:39,513 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=65, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:39,514 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=65, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178599514 2023-07-12 16:06:39,514 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots 2023-07-12 16:06:39,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:39,515 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=65, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:39,515 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=66, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:39,516 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE in 3 msec 2023-07-12 16:06:39,516 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=66, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:39,516 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=66, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED; timeout=600000, timestamp=1689178599516 2023-07-12 16:06:39,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:39,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:39,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:39,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:39,928 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:06:39,928 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:06:39,928 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:39,928 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:06:39,928 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:06:39,929 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:39,929 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:39,933 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:39,933 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:39,933 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:39,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:39,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:39,934 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:39,934 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:39,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:39,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:39,935 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testDeleteSnapshots type FLUSH 2023-07-12 16:06:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:39,935 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testDeleteSnapshots type FLUSH 2023-07-12 16:06:39,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:39,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:39,935 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:39,935 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:39,935 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:39,935 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:39,936 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:39,936 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:39,937 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:39,937 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:39,937 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:39,937 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:39,939 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:39,939 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:39,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:39,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:39,940 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:39,940 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,940 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:39,940 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:39,941 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:39,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:06:39,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd8b876[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:39,942 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:06:39,942 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,944 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,944 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:39,944 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:39,944 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:39,944 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:39,944 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:39,944 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:39,944 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:39,944 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:39,944 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:39,944 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:39,945 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. started... 2023-07-12 16:06:39,945 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2745): Flushing 8f86f373a8ffe82c020499e3ca1d249e 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:06:39,947 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:39,947 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:39,947 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:39,962 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/.tmp/f1/de4019643a9d43e98c35e30cd76fb256 2023-07-12 16:06:39,970 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/.tmp/f1/de4019643a9d43e98c35e30cd76fb256 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256 2023-07-12 16:06:39,976 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:06:39,976 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for 8f86f373a8ffe82c020499e3ca1d249e in 31ms, sequenceid=3004, compaction requested=false 2023-07-12 16:06:39,977 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testDeleteSnapshots' 2023-07-12 16:06:39,977 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 8f86f373a8ffe82c020499e3ca1d249e: 2023-07-12 16:06:39,977 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.' region-info for snapshot=snapshotName1 2023-07-12 16:06:39,977 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:39,978 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256] hfiles 2023-07-12 16:06:39,978 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256 for snapshot=snapshotName1 2023-07-12 16:06:40,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:40,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:40,384 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. completed. 2023-07-12 16:06:40,384 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:40,384 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:40,384 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:40,384 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,384 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:40,384 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:40,386 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:40,386 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,386 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:40,386 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:40,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:06:40,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:06:40,390 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:06:40,390 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:40,390 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:40,390 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:06:40,390 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:40,392 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,392 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,392 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:40,392 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:40,392 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:40,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,393 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:40,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,393 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,395 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:40,395 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:40,398 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:40,398 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:40,398 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:40,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:40,398 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:40,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/snapshotName1 because node does not exist (not an error) 2023-07-12 16:06:40,399 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:06:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:40,399 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:06:40,399 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:40,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:40,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:40,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:40,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:40,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:40,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:40,400 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:40,400 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:40,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:40,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:40,819 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:06:41,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:41,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:41,639 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testDeleteSnapshots completed 2023-07-12 16:06:41,639 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:41,640 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:41,640 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots at 1689177999514Consolidate snapshot: snapshotName1 at 1689178000399 (+885 ms)Loading Region manifests for snapshotName1 at 1689178000399Writing data manifest for snapshotName1 at 1689178000403 (+4 ms)Verifying snapshot: snapshotName1 at 1689178000810 (+407 ms)Snapshot snapshotName1 of table testDeleteSnapshots completed at 1689178001639 (+829 ms) 2023-07-12 16:06:41,641 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=66, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:41,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=66, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED in 2.1270 sec 2023-07-12 16:06:42,769 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:06:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:43,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178003570 (current time:1689178003570). 2023-07-12 16:06:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:43,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x46ff1a7c to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:43,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21bea691, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:43,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:43,579 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:43,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46ff1a7c to 127.0.0.1:55421 2023-07-12 16:06:43,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:43,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:43,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:43,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=67, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:43,583 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=67, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:43,583 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=67, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178603583 2023-07-12 16:06:43,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:43,583 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots 2023-07-12 16:06:43,584 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=67, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:43,584 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=68, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:43,585 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE in 2 msec 2023-07-12 16:06:43,585 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=68, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:43,585 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=68, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED; timeout=600000, timestamp=1689178603585 2023-07-12 16:06:43,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:43,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:43,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:43,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:43,994 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName2 2023-07-12 16:06:43,994 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName2' 2023-07-12 16:06:43,994 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:43,995 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName2' starting 'acquire' 2023-07-12 16:06:43,995 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName2', kicking off acquire phase on members. 2023-07-12 16:06:43,995 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:43,995 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:43,997 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:43,997 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:43,997 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:43,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:43,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:43,997 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:43,997 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testDeleteSnapshots type FLUSH 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testDeleteSnapshots type FLUSH 2023-07-12 16:06:43,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:06:43,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:06:43,999 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:06:43,999 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:06:43,999 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:43,999 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:43,999 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:06:44,000 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:06:44,000 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:06:44,000 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:06:44,000 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:06:44,000 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:06:44,001 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,001 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,001 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:44,002 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,002 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:44,002 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,002 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:44,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName2' on coordinator 2023-07-12 16:06:44,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@109ff109[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:44,003 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName2' starting 'in-barrier' execution. 2023-07-12 16:06:44,004 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,005 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,005 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,005 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,005 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,005 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:06:44,005 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:06:44,005 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:06:44,005 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:44,005 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:44,005 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:44,005 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:44,005 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. started... 2023-07-12 16:06:44,006 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 8f86f373a8ffe82c020499e3ca1d249e: 2023-07-12 16:06:44,006 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.' region-info for snapshot=snapshotName2 2023-07-12 16:06:44,006 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:44,006 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256] hfiles 2023-07-12 16:06:44,006 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256 for snapshot=snapshotName2 2023-07-12 16:06:44,007 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:06:44,007 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:44,007 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:06:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:44,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:44,412 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. completed. 2023-07-12 16:06:44,412 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:44,412 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:44,412 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:44,412 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,412 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:06:44,412 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:44,414 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:06:44,414 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,414 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:44,414 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:06:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName2' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:06:44,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName2', counting down latch. Waiting for 0 more 2023-07-12 16:06:44,418 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName2' execution completed 2023-07-12 16:06:44,418 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:44,418 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:44,418 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName2 2023-07-12 16:06:44,418 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:44,420 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,420 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,420 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:44,420 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:44,420 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:44,420 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,421 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,422 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:44,423 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,425 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,425 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:44,425 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:44,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:44,426 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:44,425 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:44,426 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:44,426 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,426 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:44,426 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName2 2023-07-12 16:06:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:44,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:44,426 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName2 2023-07-12 16:06:44,426 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:44,427 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:44,427 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:44,427 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:44,427 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:44,427 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:44,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:44,427 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:44,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:44,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:44,844 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName2 2023-07-12 16:06:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:45,662 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName2 of table testDeleteSnapshots completed 2023-07-12 16:06:45,662 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:45,662 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:06:45,662 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots at 1689178003583Consolidate snapshot: snapshotName2 at 1689178004426 (+843 ms)Loading Region manifests for snapshotName2 at 1689178004427 (+1 ms)Writing data manifest for snapshotName2 at 1689178004430 (+3 ms)Verifying snapshot: snapshotName2 at 1689178004837 (+407 ms)Snapshot snapshotName2 of table testDeleteSnapshots completed at 1689178005662 (+825 ms) 2023-07-12 16:06:45,663 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=68, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:45,664 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=68, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED in 2.0790 sec 2023-07-12 16:06:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:47,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:47,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178007651 (current time:1689178007651). 2023-07-12 16:06:47,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:47,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x062f2f48 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:47,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c635ab3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:47,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:47,661 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:47,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062f2f48 to 127.0.0.1:55421 2023-07-12 16:06:47,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:47,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:47,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:47,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=69, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:47,665 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=69, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:47,666 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=69, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178607666 2023-07-12 16:06:47,666 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName3 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots 2023-07-12 16:06:47,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:47,667 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=70, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:47,668 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=69, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:06:47,668 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=69, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE in 4 msec 2023-07-12 16:06:47,669 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=70, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:47,669 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=70, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED; timeout=600000, timestamp=1689178607669 2023-07-12 16:06:47,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:47,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:47,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:48,083 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName3 2023-07-12 16:06:48,083 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName3' 2023-07-12 16:06:48,083 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:48,083 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName3' starting 'acquire' 2023-07-12 16:06:48,083 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName3', kicking off acquire phase on members. 2023-07-12 16:06:48,084 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,084 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:06:48,085 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:48,085 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:48,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:48,085 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:06:48,086 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:06:48,086 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:48,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:06:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:06:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:06:48,087 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName3 from table testDeleteSnapshots type FLUSH 2023-07-12 16:06:48,087 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName3 from table testDeleteSnapshots type FLUSH 2023-07-12 16:06:48,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName3 2023-07-12 16:06:48,087 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName3' with timeout 300000ms 2023-07-12 16:06:48,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName3 2023-07-12 16:06:48,087 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:48,087 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName3' with timeout 300000ms 2023-07-12 16:06:48,088 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:48,088 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName3' starting 'acquire' stage 2023-07-12 16:06:48,088 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName3' locally acquired 2023-07-12 16:06:48,088 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName3' starting 'acquire' stage 2023-07-12 16:06:48,089 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName3' locally acquired 2023-07-12 16:06:48,089 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName3) in zk 2023-07-12 16:06:48,089 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName3) in zk 2023-07-12 16:06:48,094 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,094 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,094 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:48,095 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,095 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:48,095 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,095 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:06:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:48,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName3' on coordinator 2023-07-12 16:06:48,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2af59899[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:48,096 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName3' starting 'in-barrier' execution. 2023-07-12 16:06:48,096 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,098 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,098 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,098 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName3' received 'reached' from coordinator. 2023-07-12 16:06:48,098 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName3' received 'reached' from coordinator. 2023-07-12 16:06:48,098 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName3' locally completed 2023-07-12 16:06:48,098 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName3' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:48,098 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,098 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:48,098 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:48,098 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:48,098 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:48,098 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. started... 2023-07-12 16:06:48,098 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 8f86f373a8ffe82c020499e3ca1d249e: 2023-07-12 16:06:48,099 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.' region-info for snapshot=snapshotName3 2023-07-12 16:06:48,099 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:48,099 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256] hfiles 2023-07-12 16:06:48,099 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256 for snapshot=snapshotName3 2023-07-12 16:06:48,100 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName3' has notified controller of completion 2023-07-12 16:06:48,100 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:48,100 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName3' completed. 2023-07-12 16:06:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:48,505 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. completed. 2023-07-12 16:06:48,505 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:48,505 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:48,505 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:48,505 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,505 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName3' locally completed 2023-07-12 16:06:48,505 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName3' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:48,508 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,509 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName3' has notified controller of completion 2023-07-12 16:06:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:48,509 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:48,509 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName3' completed. 2023-07-12 16:06:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:06:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:06:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName3' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:06:48,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName3', counting down latch. Waiting for 0 more 2023-07-12 16:06:48,512 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName3' execution completed 2023-07-12 16:06:48,512 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:48,512 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:48,512 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName3 2023-07-12 16:06:48,512 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName3including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:48,514 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,514 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:48,514 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:48,514 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,514 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:48,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:48,515 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:06:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,515 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:06:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,517 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:48,517 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:06:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,520 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,520 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:48,520 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:06:48,520 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:48,520 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:48,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:48,520 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:48,520 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName3 2023-07-12 16:06:48,520 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,521 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName3 2023-07-12 16:06:48,520 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:48,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:48,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:48,521 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:06:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:48,521 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:48,521 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,521 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:48,521 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:06:48,521 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:06:48,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:48,522 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:06:48,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:48,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:48,940 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName3 2023-07-12 16:06:49,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:49,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:49,759 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName3 of table testDeleteSnapshots completed 2023-07-12 16:06:49,759 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:06:49,760 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:06:49,760 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName3 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots at 1689178007666Consolidate snapshot: snapshotName3 at 1689178008521 (+855 ms)Loading Region manifests for snapshotName3 at 1689178008521Writing data manifest for snapshotName3 at 1689178008525 (+4 ms)Verifying snapshot: snapshotName3 at 1689178008934 (+409 ms)Snapshot snapshotName3 of table testDeleteSnapshots completed at 1689178009759 (+825 ms) 2023-07-12 16:06:49,762 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=70, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:06:49,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=70, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED in 2.0950 sec 2023-07-12 16:06:51,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:51,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:51,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" type: DISABLED 2023-07-12 16:06:51,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:06:51,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName2" table: "testDeleteSnapshots" creation_time: 1689178003570 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:06:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName2 2023-07-12 16:06:51,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName3" table: "testDeleteSnapshots" creation_time: 1689178007651 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:06:51,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName3 2023-07-12 16:06:51,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testDeleteSnapshots 2023-07-12 16:06:51,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=71, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDeleteSnapshots 2023-07-12 16:06:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=71 2023-07-12 16:06:51,822 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178011822"}]},"ts":"1689178011822"} 2023-07-12 16:06:51,823 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteSnapshots, state=DISABLING in hbase:meta 2023-07-12 16:06:51,826 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testDeleteSnapshots to state=DISABLING 2023-07-12 16:06:51,826 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=71, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteSnapshots, region=8f86f373a8ffe82c020499e3ca1d249e, UNASSIGN}] 2023-07-12 16:06:51,827 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, ppid=71, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteSnapshots, region=8f86f373a8ffe82c020499e3ca1d249e, UNASSIGN 2023-07-12 16:06:51,828 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=72 updating hbase:meta row=8f86f373a8ffe82c020499e3ca1d249e, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:51,828 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178011828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178011828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178011828"}]},"ts":"1689178011828"} 2023-07-12 16:06:51,829 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=72, state=RUNNABLE; CloseRegionProcedure 8f86f373a8ffe82c020499e3ca1d249e, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:06:51,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=71 2023-07-12 16:06:51,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:51,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 8f86f373a8ffe82c020499e3ca1d249e, disabling compactions & flushes 2023-07-12 16:06:51,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:51,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:51,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. after waiting 0 ms 2023-07-12 16:06:51,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:51,986 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/recovered.edits/3009.seqid, newMaxSeqId=3009, maxSeqId=1 2023-07-12 16:06:51,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. 2023-07-12 16:06:51,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 8f86f373a8ffe82c020499e3ca1d249e: 2023-07-12 16:06:51,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:51,989 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=72 updating hbase:meta row=8f86f373a8ffe82c020499e3ca1d249e, regionState=CLOSED 2023-07-12 16:06:51,989 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178011989"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178011989"}]},"ts":"1689178011989"} 2023-07-12 16:06:51,991 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=72 2023-07-12 16:06:51,991 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=72, state=SUCCESS; CloseRegionProcedure 8f86f373a8ffe82c020499e3ca1d249e, server=jenkins-hbase5.apache.org,38637,1689177958270 in 161 msec 2023-07-12 16:06:51,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=71 2023-07-12 16:06:51,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=71, state=SUCCESS; TransitRegionStateProcedure table=testDeleteSnapshots, region=8f86f373a8ffe82c020499e3ca1d249e, UNASSIGN in 165 msec 2023-07-12 16:06:51,992 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178011992"}]},"ts":"1689178011992"} 2023-07-12 16:06:51,993 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteSnapshots, state=DISABLED in hbase:meta 2023-07-12 16:06:51,995 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testDeleteSnapshots to state=DISABLED 2023-07-12 16:06:51,996 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=71, state=SUCCESS; DisableTableProcedure table=testDeleteSnapshots in 176 msec 2023-07-12 16:06:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=71 2023-07-12 16:06:52,139 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testDeleteSnapshots completed 2023-07-12 16:06:52,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testDeleteSnapshots 2023-07-12 16:06:52,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=74, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:06:52,142 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=74, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:06:52,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=74 2023-07-12 16:06:52,143 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=74, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:06:52,147 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:52,149 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/recovered.edits] 2023-07-12 16:06:52,155 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/f1/de4019643a9d43e98c35e30cd76fb256 2023-07-12 16:06:52,158 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/recovered.edits/3009.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e/recovered.edits/3009.seqid 2023-07-12 16:06:52,159 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/8f86f373a8ffe82c020499e3ca1d249e 2023-07-12 16:06:52,159 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testDeleteSnapshots regions 2023-07-12 16:06:52,161 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=74, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:06:52,163 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testDeleteSnapshots from hbase:meta 2023-07-12 16:06:52,169 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'testDeleteSnapshots' descriptor. 2023-07-12 16:06:52,170 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=74, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:06:52,170 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'testDeleteSnapshots' from region states. 2023-07-12 16:06:52,170 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178012170"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:52,171 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:06:52,171 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8f86f373a8ffe82c020499e3ca1d249e, NAME => 'testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:06:52,171 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'testDeleteSnapshots' as deleted. 2023-07-12 16:06:52,172 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689178012171"}]},"ts":"9223372036854775807"} 2023-07-12 16:06:52,173 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table testDeleteSnapshots state from META 2023-07-12 16:06:52,175 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=74, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:06:52,176 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=74, state=SUCCESS; DeleteTableProcedure table=testDeleteSnapshots in 35 msec 2023-07-12 16:06:52,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=74 2023-07-12 16:06:52,250 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testDeleteSnapshots 2023-07-12 16:06:52,250 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testDeleteSnapshots completed 2023-07-12 16:06:52,265 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testDeleteSnapshots[0] Thread=282 (was 286), OpenFileDescriptor=538 (was 543), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=214 (was 195) - SystemLoadAverage LEAK? -, ProcessCount=160 (was 161), AvailableMemoryMB=6888 (was 7053) 2023-07-12 16:06:52,277 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testListSnapshots[0] Thread=282, OpenFileDescriptor=538, MaxFileDescriptor=60000, SystemLoadAverage=214, ProcessCount=160, AvailableMemoryMB=6888 2023-07-12 16:06:52,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testListSnapshots', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:06:52,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=75, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testListSnapshots 2023-07-12 16:06:52,280 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=75, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:06:52,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testListSnapshots" procId is: 75 2023-07-12 16:06:52,281 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=75, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:06:52,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=75 2023-07-12 16:06:52,282 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:52,283 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc empty. 2023-07-12 16:06:52,283 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:52,284 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testListSnapshots regions 2023-07-12 16:06:52,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=75 2023-07-12 16:06:52,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=75 2023-07-12 16:06:52,695 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/.tabledesc/.tableinfo.0000000001 2023-07-12 16:06:52,696 INFO [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6b686599b729501fea84a4a7c0fcc5bc, NAME => 'testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testListSnapshots', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:06:52,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=75 2023-07-12 16:06:53,102 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(866): Instantiated testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:53,103 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1604): Closing 6b686599b729501fea84a4a7c0fcc5bc, disabling compactions & flushes 2023-07-12 16:06:53,103 INFO [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1626): Closing region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:53,103 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:53,103 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1714): Acquired close lock on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. after waiting 0 ms 2023-07-12 16:06:53,103 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1724): Updates disabled for region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:53,103 INFO [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1838): Closed testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:53,103 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1558): Region close journal for 6b686599b729501fea84a4a7c0fcc5bc: 2023-07-12 16:06:53,105 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=75, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:06:53,106 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178013106"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178013106"}]},"ts":"1689178013106"} 2023-07-12 16:06:53,107 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:06:53,108 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=75, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:06:53,108 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178013108"}]},"ts":"1689178013108"} 2023-07-12 16:06:53,108 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testListSnapshots, state=ENABLING in hbase:meta 2023-07-12 16:06:53,111 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:06:53,112 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:06:53,112 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:06:53,112 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:06:53,112 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=76, ppid=75, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testListSnapshots, region=6b686599b729501fea84a4a7c0fcc5bc, ASSIGN}] 2023-07-12 16:06:53,114 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=76, ppid=75, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testListSnapshots, region=6b686599b729501fea84a4a7c0fcc5bc, ASSIGN 2023-07-12 16:06:53,114 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=76, ppid=75, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testListSnapshots, region=6b686599b729501fea84a4a7c0fcc5bc, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,38637,1689177958270; forceNewPlan=false, retain=false 2023-07-12 16:06:53,265 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:06:53,265 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=6b686599b729501fea84a4a7c0fcc5bc, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:53,266 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178013265"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178013265"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178013265"}]},"ts":"1689178013265"} 2023-07-12 16:06:53,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE; OpenRegionProcedure 6b686599b729501fea84a4a7c0fcc5bc, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:06:53,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=75 2023-07-12 16:06:53,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:53,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6b686599b729501fea84a4a7c0fcc5bc, NAME => 'testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:06:53,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testListSnapshots 6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:53,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:06:53,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:53,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:53,423 INFO [StoreOpener-6b686599b729501fea84a4a7c0fcc5bc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:53,425 DEBUG [StoreOpener-6b686599b729501fea84a4a7c0fcc5bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1 2023-07-12 16:06:53,425 DEBUG [StoreOpener-6b686599b729501fea84a4a7c0fcc5bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1 2023-07-12 16:06:53,426 INFO [StoreOpener-6b686599b729501fea84a4a7c0fcc5bc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b686599b729501fea84a4a7c0fcc5bc columnFamilyName f1 2023-07-12 16:06:53,427 INFO [StoreOpener-6b686599b729501fea84a4a7c0fcc5bc-1] regionserver.HStore(310): Store=6b686599b729501fea84a4a7c0fcc5bc/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:06:53,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:53,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:53,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:06:53,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:06:53,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 6b686599b729501fea84a4a7c0fcc5bc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71740277, jitterRate=0.06901343166828156}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:06:53,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 6b686599b729501fea84a4a7c0fcc5bc: 2023-07-12 16:06:53,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc., pid=77, masterSystemTime=1689178013418 2023-07-12 16:06:53,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:53,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:53,436 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=6b686599b729501fea84a4a7c0fcc5bc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:53,436 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178013436"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178013436"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178013436"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178013436"}]},"ts":"1689178013436"} 2023-07-12 16:06:53,437 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=77, resume processing ppid=76 2023-07-12 16:06:53,437 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; OpenRegionProcedure 6b686599b729501fea84a4a7c0fcc5bc, server=jenkins-hbase5.apache.org,38637,1689177958270 in 170 msec 2023-07-12 16:06:53,438 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=76, resume processing ppid=75 2023-07-12 16:06:53,438 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=76, ppid=75, state=SUCCESS; TransitRegionStateProcedure table=testListSnapshots, region=6b686599b729501fea84a4a7c0fcc5bc, ASSIGN in 325 msec 2023-07-12 16:06:53,439 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=75, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:06:53,439 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178013439"}]},"ts":"1689178013439"} 2023-07-12 16:06:53,440 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testListSnapshots, state=ENABLED in hbase:meta 2023-07-12 16:06:53,441 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=75, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:06:53,442 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=75, state=SUCCESS; CreateTableProcedure table=testListSnapshots in 1.1620 sec 2023-07-12 16:06:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=75 2023-07-12 16:06:54,386 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testListSnapshots, procId: 75 completed 2023-07-12 16:06:54,387 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testListSnapshots get assigned. Timeout = 60000ms 2023-07-12 16:06:54,387 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:54,390 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testListSnapshots assigned to meta. Checking AM states. 2023-07-12 16:06:54,390 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:06:54,390 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testListSnapshots assigned. 2023-07-12 16:06:55,081 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testListSnapshots' 2023-07-12 16:06:55,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:55,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178015337 (current time:1689178015337). 2023-07-12 16:06:55,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:55,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x57cc50d6 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:55,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@115f93ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:55,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:55,345 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:55,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57cc50d6 to 127.0.0.1:55421 2023-07-12 16:06:55,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:55,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:55,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:55,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=78, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:06:55,349 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=78, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:06:55,349 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=78, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178615349 2023-07-12 16:06:55,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:55,349 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testListSnapshots 2023-07-12 16:06:55,350 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=78, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:06:55,350 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=79, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:06:55,351 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=78, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE in 2 msec 2023-07-12 16:06:55,351 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=79, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:06:55,351 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=79, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED; timeout=600000, timestamp=1689178615351 2023-07-12 16:06:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:55,759 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName1 was in running list but was completed. Accepting new attempt. 2023-07-12 16:06:55,759 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:06:55,759 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:06:55,759 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:55,759 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:06:55,759 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:06:55,760 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:55,760 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:55,762 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:55,762 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:55,762 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:55,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:55,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:55,763 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:55,763 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:55,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:55,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:55,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:06:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:06:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:55,764 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testListSnapshots type FLUSH 2023-07-12 16:06:55,764 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testListSnapshots type FLUSH 2023-07-12 16:06:55,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:06:55,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:55,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:06:55,764 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:55,764 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:55,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:06:55,765 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:55,765 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:06:55,765 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:55,765 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:55,766 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:06:55,766 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:06:55,766 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:55,766 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:06:55,767 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,767 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:55,767 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:55,768 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,768 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:55,768 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,768 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:55,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:06:55,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16229d0d[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:55,769 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:06:55,770 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,771 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,771 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:55,771 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:55,771 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:55,771 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:06:55,771 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:55,771 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:55,771 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:55,771 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:55,771 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:55,771 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:55,771 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. started... 2023-07-12 16:06:55,772 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2745): Flushing 6b686599b729501fea84a4a7c0fcc5bc 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:06:55,773 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:55,773 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:55,773 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:56,182 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/.tmp/f1/ac3fd322842d473c87a4495303ed59c7 2023-07-12 16:06:56,188 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/.tmp/f1/ac3fd322842d473c87a4495303ed59c7 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7 2023-07-12 16:06:56,194 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:06:56,194 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for 6b686599b729501fea84a4a7c0fcc5bc in 422ms, sequenceid=3004, compaction requested=false 2023-07-12 16:06:56,195 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 6b686599b729501fea84a4a7c0fcc5bc: 2023-07-12 16:06:56,195 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.' region-info for snapshot=snapshotName1 2023-07-12 16:06:56,195 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:56,195 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7] hfiles 2023-07-12 16:06:56,195 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7 for snapshot=snapshotName1 2023-07-12 16:06:56,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:56,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:56,601 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. completed. 2023-07-12 16:06:56,601 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:56,601 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:06:56,601 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:06:56,601 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,601 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:06:56,601 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:06:56,603 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:06:56,603 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,603 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:56,603 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:06:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:06:56,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:06:56,607 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:06:56,607 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:06:56,607 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:06:56,607 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:06:56,607 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:06:56,608 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,609 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,609 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:56,608 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,609 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:56,609 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,610 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:56,613 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:56,613 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:06:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:56,619 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:56,619 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:56,619 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:56,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:56,620 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:56,620 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:06:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:56,620 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,620 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:56,619 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:56,620 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:06:56,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:56,620 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:06:56,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:56,621 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:56,620 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:06:56,621 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:56,621 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:56,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:06:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:06:56,621 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:06:56,622 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:06:56,622 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:56,640 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:06:57,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:57,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:57,456 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testListSnapshots completed 2023-07-12 16:06:57,456 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:57,457 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:06:57,457 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testListSnapshots at 1689178015349Consolidate snapshot: snapshotName1 at 1689178016620 (+1271 ms)Loading Region manifests for snapshotName1 at 1689178016621 (+1 ms)Writing data manifest for snapshotName1 at 1689178016625 (+4 ms)Verifying snapshot: snapshotName1 at 1689178016633 (+8 ms)Snapshot snapshotName1 of table testListSnapshots completed at 1689178017456 (+823 ms) 2023-07-12 16:06:57,458 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=79, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:06:57,459 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=79, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED in 2.1080 sec 2023-07-12 16:06:57,512 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:06:59,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:59,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:06:59,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178019420 (current time:1689178019420). 2023-07-12 16:06:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:06:59,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x01b62d67 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:06:59,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5252b1a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:06:59,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:06:59,430 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:06:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01b62d67 to 127.0.0.1:55421 2023-07-12 16:06:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:06:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:06:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:59,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=80, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:06:59,435 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=80, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:06:59,436 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=80, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178619436 2023-07-12 16:06:59,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:06:59,436 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testListSnapshots 2023-07-12 16:06:59,437 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=80, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:06:59,437 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=81, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:06:59,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=80, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE in 4 msec 2023-07-12 16:06:59,438 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=81, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:06:59,438 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=81, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED; timeout=600000, timestamp=1689178619438 2023-07-12 16:06:59,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:59,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:06:59,846 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName2 was in running list but was completed. Accepting new attempt. 2023-07-12 16:06:59,846 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName2 2023-07-12 16:06:59,846 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName2' 2023-07-12 16:06:59,846 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:59,846 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName2' starting 'acquire' 2023-07-12 16:06:59,846 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName2', kicking off acquire phase on members. 2023-07-12 16:06:59,847 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:59,847 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:59,848 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:59,848 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:59,848 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:06:59,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:59,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:06:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:06:59,848 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:59,848 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testListSnapshots type FLUSH 2023-07-12 16:06:59,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:06:59,850 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testListSnapshots type FLUSH 2023-07-12 16:06:59,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:06:59,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:06:59,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:06:59,850 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:06:59,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:06:59,850 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:59,850 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:06:59,850 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:06:59,850 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:06:59,851 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:06:59,851 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:06:59,851 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:06:59,851 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:06:59,851 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:06:59,853 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,854 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,854 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,854 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:59,854 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:06:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:06:59,854 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,854 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:06:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:06:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:06:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:06:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:06:59,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:06:59,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName2' on coordinator 2023-07-12 16:06:59,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ee19cb7[Count = 0] remaining members to acquire global barrier 2023-07-12 16:06:59,856 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName2' starting 'in-barrier' execution. 2023-07-12 16:06:59,856 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,857 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,857 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:06:59,858 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:06:59,858 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:06:59,858 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:06:59,858 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:06:59,858 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:06:59,858 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:06:59,858 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:06:59,858 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:06:59,858 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:06:59,858 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. started... 2023-07-12 16:06:59,858 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 6b686599b729501fea84a4a7c0fcc5bc: 2023-07-12 16:06:59,858 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.' region-info for snapshot=snapshotName2 2023-07-12 16:06:59,858 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:06:59,858 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7] hfiles 2023-07-12 16:06:59,859 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7 for snapshot=snapshotName2 2023-07-12 16:06:59,859 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:06:59,859 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:06:59,859 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:06:59,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:06:59,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:00,265 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. completed. 2023-07-12 16:07:00,265 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:07:00,265 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:00,265 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:00,265 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,265 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:07:00,265 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:00,268 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,268 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:07:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:00,268 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:00,268 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:07:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName2' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:07:00,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName2', counting down latch. Waiting for 0 more 2023-07-12 16:07:00,272 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName2' execution completed 2023-07-12 16:07:00,272 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:00,272 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:00,272 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName2 2023-07-12 16:07:00,272 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:00,274 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,274 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,274 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:00,274 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:00,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,274 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:00,275 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:00,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,275 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,276 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:00,277 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:00,279 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:00,279 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:00,279 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:00,279 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:00,279 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:00,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:00,279 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:00,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:00,279 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:00,279 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName2 2023-07-12 16:07:00,279 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,280 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName2 2023-07-12 16:07:00,280 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:00,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:00,280 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:00,281 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:00,281 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:00,281 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:00,281 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:00,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:00,282 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:00,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:00,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:00,698 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName2 2023-07-12 16:07:01,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:01,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:01,515 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName2 of table testListSnapshots completed 2023-07-12 16:07:01,515 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:01,516 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:01,516 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testListSnapshots at 1689178019436Consolidate snapshot: snapshotName2 at 1689178020280 (+844 ms)Loading Region manifests for snapshotName2 at 1689178020281 (+1 ms)Writing data manifest for snapshotName2 at 1689178020285 (+4 ms)Verifying snapshot: snapshotName2 at 1689178020691 (+406 ms)Snapshot snapshotName2 of table testListSnapshots completed at 1689178021515 (+824 ms) 2023-07-12 16:07:01,517 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=81, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:07:01,518 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=81, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED in 2.0800 sec 2023-07-12 16:07:03,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:03,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:03,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:03,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178023491 (current time:1689178023491). 2023-07-12 16:07:03,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:03,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x2a80f375 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:03,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d189834, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:03,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:03,499 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43516, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:03,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a80f375 to 127.0.0.1:55421 2023-07-12 16:07:03,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:03,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:03,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:03,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=82, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:07:03,504 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=82, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:07:03,504 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=82, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178623504 2023-07-12 16:07:03,504 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName3 C_M_SNAPSHOT_TABLE on table testListSnapshots 2023-07-12 16:07:03,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:03,505 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=82, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:07:03,505 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=83, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:07:03,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=82, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE in 2 msec 2023-07-12 16:07:03,506 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=83, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:07:03,506 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=83, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED; timeout=600000, timestamp=1689178623506 2023-07-12 16:07:03,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:03,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:03,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:03,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:03,915 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName3 was in running list but was completed. Accepting new attempt. 2023-07-12 16:07:03,915 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName3 2023-07-12 16:07:03,915 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName3' 2023-07-12 16:07:03,915 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:03,915 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName3' starting 'acquire' 2023-07-12 16:07:03,915 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName3', kicking off acquire phase on members. 2023-07-12 16:07:03,916 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:03,916 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:03,918 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:03,918 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:03,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:03,918 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:03,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:03,919 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:03,919 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:03,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:03,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:07:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:07:03,920 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName3 from table testListSnapshots type FLUSH 2023-07-12 16:07:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:03,920 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName3 from table testListSnapshots type FLUSH 2023-07-12 16:07:03,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName3 is still present, removing 2023-07-12 16:07:03,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName3 2023-07-12 16:07:03,920 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName3' with timeout 300000ms 2023-07-12 16:07:03,920 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:03,923 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName3' starting 'acquire' stage 2023-07-12 16:07:03,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName3 is still present, removing 2023-07-12 16:07:03,923 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName3' locally acquired 2023-07-12 16:07:03,923 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName3) in zk 2023-07-12 16:07:03,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName3 2023-07-12 16:07:03,923 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName3' with timeout 300000ms 2023-07-12 16:07:03,923 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:03,924 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName3' starting 'acquire' stage 2023-07-12 16:07:03,924 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName3' locally acquired 2023-07-12 16:07:03,924 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName3) in zk 2023-07-12 16:07:03,925 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,927 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:03,927 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:03,927 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,927 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:03,927 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,927 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:03,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName3' on coordinator 2023-07-12 16:07:03,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21193bdb[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:03,929 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName3' starting 'in-barrier' execution. 2023-07-12 16:07:03,929 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,931 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,931 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName3' received 'reached' from coordinator. 2023-07-12 16:07:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName3' received 'reached' from coordinator. 2023-07-12 16:07:03,931 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:03,931 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName3' locally completed 2023-07-12 16:07:03,931 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:07:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName3' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:03,931 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. started... 2023-07-12 16:07:03,932 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 6b686599b729501fea84a4a7c0fcc5bc: 2023-07-12 16:07:03,932 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.' region-info for snapshot=snapshotName3 2023-07-12 16:07:03,932 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:03,932 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7] hfiles 2023-07-12 16:07:03,932 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7 for snapshot=snapshotName3 2023-07-12 16:07:03,934 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName3' has notified controller of completion 2023-07-12 16:07:03,934 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:03,934 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName3' completed. 2023-07-12 16:07:04,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:04,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:04,340 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. completed. 2023-07-12 16:07:04,340 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:07:04,340 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:04,340 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:04,340 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,340 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName3' locally completed 2023-07-12 16:07:04,340 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName3' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:04,342 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,342 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName3' has notified controller of completion 2023-07-12 16:07:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:04,342 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:04,343 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName3' completed. 2023-07-12 16:07:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName3' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:07:04,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName3', counting down latch. Waiting for 0 more 2023-07-12 16:07:04,347 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName3' execution completed 2023-07-12 16:07:04,347 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:04,347 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:04,347 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName3 2023-07-12 16:07:04,347 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName3including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:04,349 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,350 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,350 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:04,350 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,350 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:04,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:04,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:04,350 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:04,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:04,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:04,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:04,351 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:04,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:04,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,353 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:04,353 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:04,360 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:04,360 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:04,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:04,360 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:04,360 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:04,360 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:04,361 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:04,361 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:04,361 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName3 2023-07-12 16:07:04,362 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName3 2023-07-12 16:07:04,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:04,361 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:04,361 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:04,362 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:04,362 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:04,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:04,363 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:04,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:04,363 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:04,363 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:07:04,363 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:04,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:04,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:04,785 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName3 2023-07-12 16:07:05,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:05,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:05,601 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName3 of table testListSnapshots completed 2023-07-12 16:07:05,602 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:07:05,602 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:07:05,602 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName3 C_M_SNAPSHOT_TABLE on table testListSnapshots at 1689178023504Consolidate snapshot: snapshotName3 at 1689178024362 (+858 ms)Loading Region manifests for snapshotName3 at 1689178024362Writing data manifest for snapshotName3 at 1689178024366 (+4 ms)Verifying snapshot: snapshotName3 at 1689178024778 (+412 ms)Snapshot snapshotName3 of table testListSnapshots completed at 1689178025601 (+823 ms) 2023-07-12 16:07:05,603 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=83, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:07:05,604 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=83, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED in 2.0980 sec 2023-07-12 16:07:07,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:07,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:07,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" table: "testListSnapshots" creation_time: 1689178015337 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:07,620 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName3" table: "testListSnapshots" creation_time: 1689178023491 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:07,620 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName2" table: "testListSnapshots" creation_time: 1689178019420 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:07,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName2 2023-07-12 16:07:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName3 2023-07-12 16:07:07,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:07:07,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testListSnapshots 2023-07-12 16:07:07,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=84, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testListSnapshots 2023-07-12 16:07:07,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=84 2023-07-12 16:07:07,639 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178027638"}]},"ts":"1689178027638"} 2023-07-12 16:07:07,639 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testListSnapshots, state=DISABLING in hbase:meta 2023-07-12 16:07:07,641 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testListSnapshots to state=DISABLING 2023-07-12 16:07:07,642 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=84, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testListSnapshots, region=6b686599b729501fea84a4a7c0fcc5bc, UNASSIGN}] 2023-07-12 16:07:07,643 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=85, ppid=84, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testListSnapshots, region=6b686599b729501fea84a4a7c0fcc5bc, UNASSIGN 2023-07-12 16:07:07,644 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=85 updating hbase:meta row=6b686599b729501fea84a4a7c0fcc5bc, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:07,644 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178027644"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178027644"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178027644"}]},"ts":"1689178027644"} 2023-07-12 16:07:07,645 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=85, state=RUNNABLE; CloseRegionProcedure 6b686599b729501fea84a4a7c0fcc5bc, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:07,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=84 2023-07-12 16:07:07,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:07:07,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 6b686599b729501fea84a4a7c0fcc5bc, disabling compactions & flushes 2023-07-12 16:07:07,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:07:07,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:07:07,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. after waiting 0 ms 2023-07-12 16:07:07,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:07:07,802 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/recovered.edits/3009.seqid, newMaxSeqId=3009, maxSeqId=1 2023-07-12 16:07:07,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. 2023-07-12 16:07:07,803 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 6b686599b729501fea84a4a7c0fcc5bc: 2023-07-12 16:07:07,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:07:07,805 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=85 updating hbase:meta row=6b686599b729501fea84a4a7c0fcc5bc, regionState=CLOSED 2023-07-12 16:07:07,805 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178027805"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178027805"}]},"ts":"1689178027805"} 2023-07-12 16:07:07,808 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=85 2023-07-12 16:07:07,808 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=85, state=SUCCESS; CloseRegionProcedure 6b686599b729501fea84a4a7c0fcc5bc, server=jenkins-hbase5.apache.org,38637,1689177958270 in 161 msec 2023-07-12 16:07:07,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=84 2023-07-12 16:07:07,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=84, state=SUCCESS; TransitRegionStateProcedure table=testListSnapshots, region=6b686599b729501fea84a4a7c0fcc5bc, UNASSIGN in 166 msec 2023-07-12 16:07:07,809 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178027809"}]},"ts":"1689178027809"} 2023-07-12 16:07:07,810 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testListSnapshots, state=DISABLED in hbase:meta 2023-07-12 16:07:07,812 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testListSnapshots to state=DISABLED 2023-07-12 16:07:07,813 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=84, state=SUCCESS; DisableTableProcedure table=testListSnapshots in 176 msec 2023-07-12 16:07:07,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=84 2023-07-12 16:07:07,960 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testListSnapshots completed 2023-07-12 16:07:07,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testListSnapshots 2023-07-12 16:07:07,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=87, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:07:07,963 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=87, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:07:07,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=87 2023-07-12 16:07:07,963 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=87, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:07:07,967 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:07:07,969 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/recovered.edits] 2023-07-12 16:07:07,975 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/f1/ac3fd322842d473c87a4495303ed59c7 2023-07-12 16:07:07,978 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/recovered.edits/3009.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc/recovered.edits/3009.seqid 2023-07-12 16:07:07,979 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/6b686599b729501fea84a4a7c0fcc5bc 2023-07-12 16:07:07,979 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testListSnapshots regions 2023-07-12 16:07:07,982 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=87, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:07:07,983 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testListSnapshots from hbase:meta 2023-07-12 16:07:07,985 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testListSnapshots' descriptor. 2023-07-12 16:07:07,986 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=87, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:07:07,986 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testListSnapshots' from region states. 2023-07-12 16:07:07,986 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178027986"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:07,988 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:07:07,988 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6b686599b729501fea84a4a7c0fcc5bc, NAME => 'testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:07,988 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testListSnapshots' as deleted. 2023-07-12 16:07:07,988 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689178027988"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:07,990 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testListSnapshots state from META 2023-07-12 16:07:07,991 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=87, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:07:07,992 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=87, state=SUCCESS; DeleteTableProcedure table=testListSnapshots in 30 msec 2023-07-12 16:07:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=87 2023-07-12 16:07:08,070 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testListSnapshots 2023-07-12 16:07:08,070 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testListSnapshots completed 2023-07-12 16:07:08,085 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testListSnapshots[0] Thread=267 (was 282), OpenFileDescriptor=522 (was 538), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=231 (was 214) - SystemLoadAverage LEAK? -, ProcessCount=160 (was 160), AvailableMemoryMB=7206 (was 6888) - AvailableMemoryMB LEAK? - 2023-07-12 16:07:08,097 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testRestoreSnapshot[1] Thread=267, OpenFileDescriptor=522, MaxFileDescriptor=60000, SystemLoadAverage=231, ProcessCount=160, AvailableMemoryMB=7204 2023-07-12 16:07:08,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testRestoreSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:07:08,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=88, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:08,102 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=88, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:07:08,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot" procId is: 88 2023-07-12 16:07:08,103 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=88, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:07:08,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-07-12 16:07:08,105 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:08,105 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 empty. 2023-07-12 16:07:08,106 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:08,106 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot regions 2023-07-12 16:07:08,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-07-12 16:07:08,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-07-12 16:07:08,517 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000001 2023-07-12 16:07:08,518 INFO [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:07:08,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-07-12 16:07:08,924 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:08,924 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1604): Closing 79bcb320657f5226c4593c04f9f0b6c6, disabling compactions & flushes 2023-07-12 16:07:08,924 INFO [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:08,924 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:08,924 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. after waiting 0 ms 2023-07-12 16:07:08,924 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:08,925 INFO [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1838): Closed testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:08,925 DEBUG [RegionOpenAndInit-testRestoreSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:08,927 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=88, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:07:08,928 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178028927"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178028927"}]},"ts":"1689178028927"} 2023-07-12 16:07:08,928 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:07:08,929 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=88, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:07:08,929 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178028929"}]},"ts":"1689178028929"} 2023-07-12 16:07:08,930 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:07:08,932 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:07:08,933 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:07:08,933 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:07:08,933 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:07:08,933 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=89, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN}] 2023-07-12 16:07:08,934 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=89, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN 2023-07-12 16:07:08,935 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=89, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,38637,1689177958270; forceNewPlan=false, retain=false 2023-07-12 16:07:09,085 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:07:09,086 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=89 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:09,086 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178029086"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178029086"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178029086"}]},"ts":"1689178029086"} 2023-07-12 16:07:09,088 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=90, ppid=89, state=RUNNABLE; OpenRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:09,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-07-12 16:07:09,242 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:09,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:07:09,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:09,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:09,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:09,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:09,244 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:09,246 DEBUG [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1 2023-07-12 16:07:09,246 DEBUG [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1 2023-07-12 16:07:09,246 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 79bcb320657f5226c4593c04f9f0b6c6 columnFamilyName f1 2023-07-12 16:07:09,247 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] regionserver.HStore(310): Store=79bcb320657f5226c4593c04f9f0b6c6/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:07:09,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:09,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:09,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:09,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:07:09,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 79bcb320657f5226c4593c04f9f0b6c6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65053767, jitterRate=-0.030623331665992737}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:07:09,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:09,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6., pid=90, masterSystemTime=1689178029239 2023-07-12 16:07:09,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:09,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:09,255 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=89 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:09,255 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178029255"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178029255"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178029255"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178029255"}]},"ts":"1689178029255"} 2023-07-12 16:07:09,257 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=90, resume processing ppid=89 2023-07-12 16:07:09,257 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, ppid=89, state=SUCCESS; OpenRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,38637,1689177958270 in 169 msec 2023-07-12 16:07:09,258 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=89, resume processing ppid=88 2023-07-12 16:07:09,258 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=89, ppid=88, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN in 324 msec 2023-07-12 16:07:09,259 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=88, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:07:09,259 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178029259"}]},"ts":"1689178029259"} 2023-07-12 16:07:09,259 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:07:09,262 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=88, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:07:09,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=88, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot in 1.1610 sec 2023-07-12 16:07:10,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=88 2023-07-12 16:07:10,208 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot, procId: 88 completed 2023-07-12 16:07:10,208 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testRestoreSnapshot get assigned. Timeout = 60000ms 2023-07-12 16:07:10,208 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:07:10,211 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testRestoreSnapshot assigned to meta. Checking AM states. 2023-07-12 16:07:10,211 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:07:10,211 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testRestoreSnapshot assigned. 2023-07-12 16:07:10,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.CallRunner(144): callId: 15100 service: ClientService methodName: Mutate size: 145 connection: 172.31.10.131:43266 deadline: 1689178090212, exception=org.apache.hadoop.hbase.NotServingRegionException: testRestoreSnapshot,,1689177963051.7def597ffc2b8cfcdf545c2c6ab26304. is not online on jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:11,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:11,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178031457 (current time:1689178031457). 2023-07-12 16:07:11,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:11,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x2cb6c9ba to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:11,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b5507fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:11,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:11,464 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:51504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:11,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cb6c9ba to 127.0.0.1:55421 2023-07-12 16:07:11,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:11,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:11,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:11,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:11,468 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=91, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:11,468 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=91, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178631468 2023-07-12 16:07:11,468 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot 2023-07-12 16:07:11,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:11,469 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=91, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:11,470 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:07:11,470 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE in 2 msec 2023-07-12 16:07:11,470 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=92, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:07:11,471 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=92, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED; timeout=600000, timestamp=1689178631471 2023-07-12 16:07:11,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:11,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:11,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:11,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:11,878 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName1 was in running list but was completed. Accepting new attempt. 2023-07-12 16:07:11,879 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:07:11,879 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:07:11,879 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:11,879 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:07:11,879 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:07:11,879 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:11,879 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:11,882 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:11,882 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:11,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:11,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:11,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:11,882 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:11,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:11,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:11,882 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:11,883 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testRestoreSnapshot type FLUSH 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:11,883 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testRestoreSnapshot type FLUSH 2023-07-12 16:07:11,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:07:11,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:07:11,884 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:07:11,884 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:11,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:07:11,884 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:07:11,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:07:11,884 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:07:11,884 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:07:11,884 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:11,884 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:07:11,885 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:07:11,885 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:07:11,885 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:07:11,886 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,886 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:11,886 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:11,887 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,887 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:11,887 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,887 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:11,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:07:11,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272e4e31[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:11,888 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:07:11,889 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,890 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,890 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,890 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:07:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:11,890 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:11,890 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:11,890 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:11,890 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:11,890 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:07:11,890 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:11,890 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:07:11,890 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. started... 2023-07-12 16:07:11,890 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:11,891 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2745): Flushing 79bcb320657f5226c4593c04f9f0b6c6 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:07:11,892 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:07:11,892 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:11,892 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:07:12,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:12,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:12,302 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/.tmp/f1/eba7aa12ec1c4122ba56323741402f9e 2023-07-12 16:07:12,309 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/.tmp/f1/eba7aa12ec1c4122ba56323741402f9e as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e 2023-07-12 16:07:12,314 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:07:12,315 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for 79bcb320657f5226c4593c04f9f0b6c6 in 424ms, sequenceid=3004, compaction requested=false 2023-07-12 16:07:12,315 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot' 2023-07-12 16:07:12,316 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:12,316 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.' region-info for snapshot=snapshotName1 2023-07-12 16:07:12,316 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:12,316 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e] hfiles 2023-07-12 16:07:12,316 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e for snapshot=snapshotName1 2023-07-12 16:07:12,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:12,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:12,721 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. completed. 2023-07-12 16:07:12,721 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:12,721 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:12,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:12,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:07:12,722 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:12,724 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,724 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:07:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:12,724 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:12,724 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:07:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:12,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:12,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:12,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:07:12,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:07:12,728 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:07:12,728 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:12,728 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:12,728 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:07:12,728 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:12,730 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,730 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:12,730 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,730 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:12,730 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,730 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:12,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:12,731 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:12,732 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:12,732 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:12,735 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:12,735 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:12,735 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:12,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:12,735 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:12,735 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:12,735 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:07:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:12,736 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:07:12,735 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,735 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:12,736 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:12,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:12,735 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:12,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:12,736 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:12,736 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:12,736 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:12,736 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:12,736 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:12,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:12,736 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:13,152 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:07:13,329 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:07:13,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:13,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:13,972 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testRestoreSnapshot completed 2023-07-12 16:07:13,972 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:13,973 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:13,973 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot at 1689178031468Consolidate snapshot: snapshotName1 at 1689178032735 (+1267 ms)Loading Region manifests for snapshotName1 at 1689178032736 (+1 ms)Writing data manifest for snapshotName1 at 1689178032739 (+3 ms)Verifying snapshot: snapshotName1 at 1689178033145 (+406 ms)Snapshot snapshotName1 of table testRestoreSnapshot completed at 1689178033972 (+827 ms) 2023-07-12 16:07:13,974 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=92, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:07:13,975 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED in 2.5050 sec 2023-07-12 16:07:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:15,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:15,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178035541 (current time:1689178035541). 2023-07-12 16:07:15,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:15,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x1eb86eca to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:15,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d99fbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:15,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:15,548 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:51506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:15,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1eb86eca to 127.0.0.1:55421 2023-07-12 16:07:15,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:15,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:15,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:15,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=93, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:15,552 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=93, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:15,553 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=93, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178635553 2023-07-12 16:07:15,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:15,553 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot 2023-07-12 16:07:15,554 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=93, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:15,554 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=94, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:07:15,554 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE in 3 msec 2023-07-12 16:07:15,555 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=94, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:07:15,555 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=94, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED; timeout=600000, timestamp=1689178635555 2023-07-12 16:07:15,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:15,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:15,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:15,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:15,963 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName2 was in running list but was completed. Accepting new attempt. 2023-07-12 16:07:15,963 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName2 2023-07-12 16:07:15,963 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName2' 2023-07-12 16:07:15,963 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:15,963 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName2' starting 'acquire' 2023-07-12 16:07:15,963 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName2', kicking off acquire phase on members. 2023-07-12 16:07:15,964 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:15,964 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:15,965 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:15,965 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:15,965 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:15,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:15,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:15,967 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:15,967 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:15,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:15,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:15,968 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testRestoreSnapshot type FLUSH 2023-07-12 16:07:15,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:15,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:15,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:07:15,969 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testRestoreSnapshot type FLUSH 2023-07-12 16:07:15,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:07:15,969 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:07:15,969 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:15,970 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:07:15,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:07:15,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:07:15,970 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:07:15,970 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:07:15,970 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:15,970 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:07:15,971 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:07:15,971 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:07:15,971 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:07:15,973 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:15,973 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:15,973 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,973 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,973 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:15,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:15,973 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,974 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:15,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:15,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName2' on coordinator 2023-07-12 16:07:15,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2566652c[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:15,975 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName2' starting 'in-barrier' execution. 2023-07-12 16:07:15,975 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,977 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,977 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:15,977 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:07:15,977 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:15,977 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:15,977 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:07:15,977 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:07:15,977 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:15,977 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:15,977 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:15,977 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:15,978 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. started... 2023-07-12 16:07:15,978 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:15,978 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.' region-info for snapshot=snapshotName2 2023-07-12 16:07:15,978 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:15,978 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e] hfiles 2023-07-12 16:07:15,978 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e for snapshot=snapshotName2 2023-07-12 16:07:15,979 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:07:15,979 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:15,979 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:07:16,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:16,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:16,387 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. completed. 2023-07-12 16:07:16,388 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:16,388 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:16,388 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:16,388 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,388 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:07:16,388 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:16,390 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:07:16,390 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,390 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:16,390 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:07:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName2' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:07:16,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName2', counting down latch. Waiting for 0 more 2023-07-12 16:07:16,394 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName2' execution completed 2023-07-12 16:07:16,394 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:16,394 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:16,395 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName2 2023-07-12 16:07:16,395 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:16,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,399 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,400 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,400 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,400 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:16,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:16,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,401 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:16,402 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:16,402 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:16,405 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:16,405 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:16,405 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:16,405 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName2 2023-07-12 16:07:16,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:16,405 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:16,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:16,405 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:16,405 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:16,405 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:16,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:16,406 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:16,405 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName2 2023-07-12 16:07:16,406 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:16,406 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:16,406 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:16,406 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:16,406 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:16,406 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:16,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:16,407 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:16,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:16,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:16,822 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName2 2023-07-12 16:07:17,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:17,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:17,639 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName2 of table testRestoreSnapshot completed 2023-07-12 16:07:17,639 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:17,640 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:17,640 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot at 1689178035553Consolidate snapshot: snapshotName2 at 1689178036405 (+852 ms)Loading Region manifests for snapshotName2 at 1689178036406 (+1 ms)Writing data manifest for snapshotName2 at 1689178036409 (+3 ms)Verifying snapshot: snapshotName2 at 1689178036816 (+407 ms)Snapshot snapshotName2 of table testRestoreSnapshot completed at 1689178037639 (+823 ms) 2023-07-12 16:07:17,641 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=94, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED 2023-07-12 16:07:17,642 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=94, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=SHARED in 2.0880 sec 2023-07-12 16:07:19,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:19,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:19,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testRestoreSnapshot 2023-07-12 16:07:19,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=95, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:19,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=95 2023-07-12 16:07:19,629 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178039629"}]},"ts":"1689178039629"} 2023-07-12 16:07:19,629 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:07:19,631 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot to state=DISABLING 2023-07-12 16:07:19,633 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=96, ppid=95, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN}] 2023-07-12 16:07:19,634 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=95, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN 2023-07-12 16:07:19,634 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:19,634 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178039634"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178039634"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178039634"}]},"ts":"1689178039634"} 2023-07-12 16:07:19,635 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=97, ppid=96, state=RUNNABLE; CloseRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=95 2023-07-12 16:07:19,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:19,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 79bcb320657f5226c4593c04f9f0b6c6, disabling compactions & flushes 2023-07-12 16:07:19,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:19,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:19,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. after waiting 0 ms 2023-07-12 16:07:19,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:19,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/recovered.edits/3008.seqid, newMaxSeqId=3008, maxSeqId=1 2023-07-12 16:07:19,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:19,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:19,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:19,795 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=CLOSED 2023-07-12 16:07:19,795 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178039795"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178039795"}]},"ts":"1689178039795"} 2023-07-12 16:07:19,797 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=97, resume processing ppid=96 2023-07-12 16:07:19,797 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=96, state=SUCCESS; CloseRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,38637,1689177958270 in 161 msec 2023-07-12 16:07:19,797 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=96, resume processing ppid=95 2023-07-12 16:07:19,798 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=95, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN in 166 msec 2023-07-12 16:07:19,798 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178039798"}]},"ts":"1689178039798"} 2023-07-12 16:07:19,799 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:07:19,801 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot to state=DISABLED 2023-07-12 16:07:19,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=95, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot in 175 msec 2023-07-12 16:07:19,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=95 2023-07-12 16:07:19,949 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:07:19,957 INFO [RPCClient-NioEventLoopGroup-6-5] client.RawAsyncHBaseAdmin(1961): Taking restore-failsafe snapshot: hbase-failsafe-snapshotName1-1689178039957 2023-07-12 16:07:19,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178039958 (current time:1689178039958). 2023-07-12 16:07:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:19,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x3f510968 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:19,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dcf48af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:19,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:19,965 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:56792, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:19,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f510968 to 127.0.0.1:55421 2023-07-12 16:07:19,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:19,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:19,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=98, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:19,969 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=98, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:19,969 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=98, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178639969 2023-07-12 16:07:19,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:19,969 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snapshotName1-1689178039957 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot 2023-07-12 16:07:19,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:19,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:20,377 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=DISABLED ttl=0 } 2023-07-12 16:07:20,378 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-07-12 16:07:20,378 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-07-12 16:07:20,380 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e] hfiles 2023-07-12 16:07:20,380 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e 2023-07-12 16:07:20,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:20,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:20,785 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=DISABLED ttl=0 } as finished. 2023-07-12 16:07:20,786 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snapshotName1-1689178039957 2023-07-12 16:07:20,786 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/hbase-failsafe-snapshotName1-1689178039957 2023-07-12 16:07:21,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:21,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:21,201 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/hbase-failsafe-snapshotName1-1689178039957 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/hbase-failsafe-snapshotName1-1689178039957 2023-07-12 16:07:22,019 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snapshotName1-1689178039957 of table testRestoreSnapshot completed 2023-07-12 16:07:22,019 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/hbase-failsafe-snapshotName1-1689178039957 2023-07-12 16:07:22,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:22,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:22,019 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/hbase-failsafe-snapshotName1-1689178039957 2023-07-12 16:07:22,019 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snapshotName1-1689178039957 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot at 1689178039969Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snapshotName1-1689178039957 table=testRestoreSnapshot type=DISABLED ttl=0 } at 1689178040377 (+408 ms)Consolidate snapshot: hbase-failsafe-snapshotName1-1689178039957 at 1689178040786 (+409 ms)Loading Region manifests for hbase-failsafe-snapshotName1-1689178039957 at 1689178040786Writing data manifest for hbase-failsafe-snapshotName1-1689178039957 at 1689178040789 (+3 ms)Verifying snapshot: hbase-failsafe-snapshotName1-1689178039957 at 1689178041195 (+406 ms)Snapshot hbase-failsafe-snapshotName1-1689178039957 of table testRestoreSnapshot completed at 1689178042019 (+824 ms) 2023-07-12 16:07:22,020 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=98, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE 2023-07-12 16:07:22,021 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=98, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot, type=EXCLUSIVE in 2.0530 sec 2023-07-12 16:07:22,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=99, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot snapshot=name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689178031457 type: FLUSH version: 2 ttl: 0 ) 2023-07-12 16:07:22,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(880): Restore snapshot=snapshotName1 as table=testRestoreSnapshot 2023-07-12 16:07:22,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=99 2023-07-12 16:07:22,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=99 2023-07-12 16:07:22,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=99 2023-07-12 16:07:22,436 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000002 2023-07-12 16:07:22,437 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000001 2023-07-12 16:07:22,437 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000002 2023-07-12 16:07:22,438 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(384): Starting restore snapshot={ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:22,440 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689178031457 type: FLUSH version: 2 ttl: 0 2023-07-12 16:07:22,440 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot 2023-07-12 16:07:22,443 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 1 regions for table=testRestoreSnapshot 2023-07-12 16:07:22,443 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:22,445 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689178031457 type: FLUSH version: 2 ttl: 0 2023-07-12 16:07:22,445 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178042445"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:22,447 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:07:22,447 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:22,447 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178042446"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178042446"}]},"ts":"1689178042446"} 2023-07-12 16:07:22,447 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:07:22,447 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 1 regions to Meta 2023-07-12 16:07:22,447 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:22,448 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(457): Restore snapshot={ ss=snapshotName1 table=testRestoreSnapshot type=FLUSH ttl=0 } on table=testRestoreSnapshot completed! 2023-07-12 16:07:22,449 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=99, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot snapshot=name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689178031457 type: FLUSH version: 2 ttl: 0 ) in 422 msec 2023-07-12 16:07:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=99 2023-07-12 16:07:22,650 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin(1985): Deleting restore-failsafe snapshot: hbase-failsafe-snapshotName1-1689178039957 2023-07-12 16:07:22,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "hbase-failsafe-snapshotName1-1689178039957" type: DISABLED 2023-07-12 16:07:22,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snapshotName1-1689178039957 2023-07-12 16:07:22,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$10(2385): Client=jenkins//172.31.10.131 enable testRestoreSnapshot 2023-07-12 16:07:22,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=100, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:22,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=100 2023-07-12 16:07:22,656 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot 2023-07-12 16:07:22,657 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178042657"}]},"ts":"1689178042657"} 2023-07-12 16:07:22,657 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:07:22,659 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-12 16:07:22,659 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=100, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN}] 2023-07-12 16:07:22,660 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=101, ppid=100, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN 2023-07-12 16:07:22,661 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=101, ppid=100, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-07-12 16:07:22,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=100 2023-07-12 16:07:22,811 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:07:22,811 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:07:22,811 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:07:22,811 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:07:22,812 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=101 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:22,812 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178042812"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178042812"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178042812"}]},"ts":"1689178042812"} 2023-07-12 16:07:22,813 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=101, state=RUNNABLE; OpenRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:22,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:22,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:07:22,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:22,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:22,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:22,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:22,969 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:22,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=100 2023-07-12 16:07:22,970 DEBUG [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1 2023-07-12 16:07:22,970 DEBUG [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1 2023-07-12 16:07:22,971 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 79bcb320657f5226c4593c04f9f0b6c6 columnFamilyName f1 2023-07-12 16:07:22,976 DEBUG [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] regionserver.HStore(539): loaded hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e 2023-07-12 16:07:22,976 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] regionserver.HStore(310): Store=79bcb320657f5226c4593c04f9f0b6c6/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:07:22,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:22,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:22,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:22,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 79bcb320657f5226c4593c04f9f0b6c6; next sequenceid=3009; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70722823, jitterRate=0.05385218560695648}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:07:22,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:22,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6., pid=102, masterSystemTime=1689178042964 2023-07-12 16:07:22,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:22,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:22,983 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=101 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=OPEN, openSeqNum=3009, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:22,983 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178042983"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178042983"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178042983"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178042983"}]},"ts":"1689178042983"} 2023-07-12 16:07:22,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=101 2023-07-12 16:07:22,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=101, state=SUCCESS; OpenRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,38637,1689177958270 in 171 msec 2023-07-12 16:07:22,986 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=100 2023-07-12 16:07:22,986 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=100, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN in 326 msec 2023-07-12 16:07:22,986 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178042986"}]},"ts":"1689178042986"} 2023-07-12 16:07:22,987 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:07:22,989 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot' was successfully enabled. 2023-07-12 16:07:22,991 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=100, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot in 336 msec 2023-07-12 16:07:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=100 2023-07-12 16:07:23,280 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: ENABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:07:23,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testRestoreSnapshot 2023-07-12 16:07:23,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=103, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:23,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=103 2023-07-12 16:07:23,301 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178043301"}]},"ts":"1689178043301"} 2023-07-12 16:07:23,302 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:07:23,303 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot to state=DISABLING 2023-07-12 16:07:23,304 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=103, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN}] 2023-07-12 16:07:23,305 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=104, ppid=103, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN 2023-07-12 16:07:23,305 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=104 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:23,306 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178043305"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178043305"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178043305"}]},"ts":"1689178043305"} 2023-07-12 16:07:23,306 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=105, ppid=104, state=RUNNABLE; CloseRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:23,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=103 2023-07-12 16:07:23,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:23,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 79bcb320657f5226c4593c04f9f0b6c6, disabling compactions & flushes 2023-07-12 16:07:23,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:23,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:23,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. after waiting 0 ms 2023-07-12 16:07:23,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:23,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/recovered.edits/3011.seqid, newMaxSeqId=3011, maxSeqId=3008 2023-07-12 16:07:23,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:23,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:23,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:23,465 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=104 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=CLOSED 2023-07-12 16:07:23,466 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178043465"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178043465"}]},"ts":"1689178043465"} 2023-07-12 16:07:23,467 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=105, resume processing ppid=104 2023-07-12 16:07:23,467 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=105, ppid=104, state=SUCCESS; CloseRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,38637,1689177958270 in 160 msec 2023-07-12 16:07:23,468 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=103 2023-07-12 16:07:23,468 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=103, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN in 163 msec 2023-07-12 16:07:23,468 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178043468"}]},"ts":"1689178043468"} 2023-07-12 16:07:23,469 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:07:23,471 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot to state=DISABLED 2023-07-12 16:07:23,472 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=103, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot in 173 msec 2023-07-12 16:07:23,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=103 2023-07-12 16:07:23,620 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:07:23,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=106, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot snapshot=name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689178035541 type: FLUSH version: 2 ttl: 0 ) 2023-07-12 16:07:23,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(880): Restore snapshot=snapshotName2 as table=testRestoreSnapshot 2023-07-12 16:07:23,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=106 2023-07-12 16:07:23,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=106 2023-07-12 16:07:23,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=106 2023-07-12 16:07:24,044 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000003 2023-07-12 16:07:24,045 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000002 2023-07-12 16:07:24,045 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/.tabledesc/.tableinfo.0000000003 2023-07-12 16:07:24,046 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(384): Starting restore snapshot={ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:24,048 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689178035541 type: FLUSH version: 2 ttl: 0 2023-07-12 16:07:24,048 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot 2023-07-12 16:07:24,051 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 1 regions for table=testRestoreSnapshot 2023-07-12 16:07:24,051 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:24,053 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689178035541 type: FLUSH version: 2 ttl: 0 2023-07-12 16:07:24,053 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178044053"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:24,055 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:07:24,055 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:24,055 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178044054"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178044054"}]},"ts":"1689178044054"} 2023-07-12 16:07:24,055 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:07:24,056 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 1 regions to Meta 2023-07-12 16:07:24,056 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:24,056 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(457): Restore snapshot={ ss=snapshotName2 table=testRestoreSnapshot type=FLUSH ttl=0 } on table=testRestoreSnapshot completed! 2023-07-12 16:07:24,057 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot snapshot=name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689178035541 type: FLUSH version: 2 ttl: 0 ) in 423 msec 2023-07-12 16:07:24,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=106 2023-07-12 16:07:24,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$10(2385): Client=jenkins//172.31.10.131 enable testRestoreSnapshot 2023-07-12 16:07:24,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=107, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:24,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=107 2023-07-12 16:07:24,263 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot 2023-07-12 16:07:24,263 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178044263"}]},"ts":"1689178044263"} 2023-07-12 16:07:24,264 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:07:24,266 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-12 16:07:24,266 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=108, ppid=107, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN}] 2023-07-12 16:07:24,267 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=107, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN 2023-07-12 16:07:24,267 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=108, ppid=107, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-07-12 16:07:24,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=107 2023-07-12 16:07:24,418 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:07:24,418 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:07:24,418 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:07:24,418 DEBUG [jenkins-hbase5:39879] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:07:24,419 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:24,419 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178044419"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178044419"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178044419"}]},"ts":"1689178044419"} 2023-07-12 16:07:24,420 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=109, ppid=108, state=RUNNABLE; OpenRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:07:24,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:24,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:07:24,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:24,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:24,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:24,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:24,578 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:24,579 DEBUG [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1 2023-07-12 16:07:24,579 DEBUG [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1 2023-07-12 16:07:24,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=107 2023-07-12 16:07:24,579 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 79bcb320657f5226c4593c04f9f0b6c6 columnFamilyName f1 2023-07-12 16:07:24,585 DEBUG [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] regionserver.HStore(539): loaded hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e 2023-07-12 16:07:24,585 INFO [StoreOpener-79bcb320657f5226c4593c04f9f0b6c6-1] regionserver.HStore(310): Store=79bcb320657f5226c4593c04f9f0b6c6/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:07:24,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:24,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:24,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:24,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 79bcb320657f5226c4593c04f9f0b6c6; next sequenceid=3012; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71060938, jitterRate=0.05889049172401428}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:07:24,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:24,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6., pid=109, masterSystemTime=1689178044571 2023-07-12 16:07:24,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:24,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:24,594 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=OPEN, openSeqNum=3012, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:24,594 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178044594"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178044594"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178044594"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178044594"}]},"ts":"1689178044594"} 2023-07-12 16:07:24,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=109, resume processing ppid=108 2023-07-12 16:07:24,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=108, state=SUCCESS; OpenRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,33647,1689177958228 in 175 msec 2023-07-12 16:07:24,597 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=108, resume processing ppid=107 2023-07-12 16:07:24,597 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=107, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, ASSIGN in 330 msec 2023-07-12 16:07:24,598 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178044598"}]},"ts":"1689178044598"} 2023-07-12 16:07:24,598 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:07:24,601 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot' was successfully enabled. 2023-07-12 16:07:24,602 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=107, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot in 341 msec 2023-07-12 16:07:24,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=107 2023-07-12 16:07:24,890 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: ENABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:07:24,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38637] ipc.CallRunner(144): callId: 18105 service: ClientService methodName: Scan size: 143 connection: 172.31.10.131:41838 deadline: 1689178104890, exception=org.apache.hadoop.hbase.NotServingRegionException: testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. is not online on jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:25,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" table: "testRestoreSnapshot" creation_time: 1689178031457 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:25,015 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName2" table: "testRestoreSnapshot" creation_time: 1689178035541 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:25,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:07:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName2 2023-07-12 16:07:25,020 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testRestoreSnapshot 2023-07-12 16:07:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=110, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:25,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=110 2023-07-12 16:07:25,025 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178045024"}]},"ts":"1689178045024"} 2023-07-12 16:07:25,026 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:07:25,028 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot to state=DISABLING 2023-07-12 16:07:25,029 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=111, ppid=110, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN}] 2023-07-12 16:07:25,030 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=110, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN 2023-07-12 16:07:25,031 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:25,032 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178045031"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178045031"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178045031"}]},"ts":"1689178045031"} 2023-07-12 16:07:25,033 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=111, state=RUNNABLE; CloseRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:07:25,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=110 2023-07-12 16:07:25,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:25,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 79bcb320657f5226c4593c04f9f0b6c6, disabling compactions & flushes 2023-07-12 16:07:25,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:25,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:25,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. after waiting 0 ms 2023-07-12 16:07:25,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:25,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/recovered.edits/3014.seqid, newMaxSeqId=3014, maxSeqId=3011 2023-07-12 16:07:25,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6. 2023-07-12 16:07:25,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 79bcb320657f5226c4593c04f9f0b6c6: 2023-07-12 16:07:25,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:25,194 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=79bcb320657f5226c4593c04f9f0b6c6, regionState=CLOSED 2023-07-12 16:07:25,194 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178045194"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178045194"}]},"ts":"1689178045194"} 2023-07-12 16:07:25,196 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=111 2023-07-12 16:07:25,196 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=111, state=SUCCESS; CloseRegionProcedure 79bcb320657f5226c4593c04f9f0b6c6, server=jenkins-hbase5.apache.org,33647,1689177958228 in 162 msec 2023-07-12 16:07:25,196 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=111, resume processing ppid=110 2023-07-12 16:07:25,197 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=110, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot, region=79bcb320657f5226c4593c04f9f0b6c6, UNASSIGN in 167 msec 2023-07-12 16:07:25,197 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178045197"}]},"ts":"1689178045197"} 2023-07-12 16:07:25,198 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:07:25,199 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot to state=DISABLED 2023-07-12 16:07:25,200 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=110, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot in 179 msec 2023-07-12 16:07:25,323 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:07:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=110 2023-07-12 16:07:25,340 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:07:25,341 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testRestoreSnapshot 2023-07-12 16:07:25,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=113, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:25,343 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=113, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=113 2023-07-12 16:07:25,344 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=113, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:25,348 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:25,350 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/recovered.edits] 2023-07-12 16:07:25,355 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/f1/eba7aa12ec1c4122ba56323741402f9e 2023-07-12 16:07:25,359 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/recovered.edits/3014.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6/recovered.edits/3014.seqid 2023-07-12 16:07:25,360 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testRestoreSnapshot/79bcb320657f5226c4593c04f9f0b6c6 2023-07-12 16:07:25,360 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot regions 2023-07-12 16:07:25,362 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=113, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:25,365 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRestoreSnapshot from hbase:meta 2023-07-12 16:07:25,366 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot' descriptor. 2023-07-12 16:07:25,367 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=113, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:25,367 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot' from region states. 2023-07-12 16:07:25,368 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178045367"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:25,369 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:07:25,369 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 79bcb320657f5226c4593c04f9f0b6c6, NAME => 'testRestoreSnapshot,,1689178028100.79bcb320657f5226c4593c04f9f0b6c6.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:25,369 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot' as deleted. 2023-07-12 16:07:25,369 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689178045369"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:25,371 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot state from META 2023-07-12 16:07:25,373 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=113, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot 2023-07-12 16:07:25,373 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=113, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot in 32 msec 2023-07-12 16:07:25,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=113 2023-07-12 16:07:25,450 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testRestoreSnapshot 2023-07-12 16:07:25,450 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testRestoreSnapshot completed 2023-07-12 16:07:25,463 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testRestoreSnapshot[1] Thread=271 (was 267) - Thread LEAK? -, OpenFileDescriptor=518 (was 522), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=181 (was 231), ProcessCount=161 (was 160) - ProcessCount LEAK? -, AvailableMemoryMB=7045 (was 7204) 2023-07-12 16:07:25,474 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testTakeSnapshot[1] Thread=271, OpenFileDescriptor=518, MaxFileDescriptor=60000, SystemLoadAverage=181, ProcessCount=161, AvailableMemoryMB=7044 2023-07-12 16:07:25,476 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testTakeSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:07:25,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=114, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testTakeSnapshot 2023-07-12 16:07:25,477 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=114, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:07:25,478 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testTakeSnapshot" procId is: 114 2023-07-12 16:07:25,478 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=114, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:07:25,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=114 2023-07-12 16:07:25,480 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:25,480 DEBUG [HFileArchiver-14] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b empty. 2023-07-12 16:07:25,481 DEBUG [HFileArchiver-14] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:25,481 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testTakeSnapshot regions 2023-07-12 16:07:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=114 2023-07-12 16:07:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=114 2023-07-12 16:07:25,890 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/.tabledesc/.tableinfo.0000000001 2023-07-12 16:07:25,892 INFO [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => afccb6ca19c037f46d82ff643517b50b, NAME => 'testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testTakeSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:07:26,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=114 2023-07-12 16:07:26,298 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(866): Instantiated testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:26,298 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1604): Closing afccb6ca19c037f46d82ff643517b50b, disabling compactions & flushes 2023-07-12 16:07:26,298 INFO [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1626): Closing region testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:26,298 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:26,298 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. after waiting 0 ms 2023-07-12 16:07:26,298 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:26,298 INFO [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1838): Closed testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:26,298 DEBUG [RegionOpenAndInit-testTakeSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for afccb6ca19c037f46d82ff643517b50b: 2023-07-12 16:07:26,300 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=114, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:07:26,301 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689178046301"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178046301"}]},"ts":"1689178046301"} 2023-07-12 16:07:26,302 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:07:26,303 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=114, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:07:26,303 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178046303"}]},"ts":"1689178046303"} 2023-07-12 16:07:26,303 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testTakeSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:07:26,308 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:07:26,308 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:07:26,309 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:07:26,309 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:07:26,309 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=114, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTakeSnapshot, region=afccb6ca19c037f46d82ff643517b50b, ASSIGN}] 2023-07-12 16:07:26,310 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=115, ppid=114, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTakeSnapshot, region=afccb6ca19c037f46d82ff643517b50b, ASSIGN 2023-07-12 16:07:26,311 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=115, ppid=114, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTakeSnapshot, region=afccb6ca19c037f46d82ff643517b50b, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,38637,1689177958270; forceNewPlan=false, retain=false 2023-07-12 16:07:26,461 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:07:26,462 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=afccb6ca19c037f46d82ff643517b50b, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:26,462 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689178046462"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178046462"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178046462"}]},"ts":"1689178046462"} 2023-07-12 16:07:26,463 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=115, state=RUNNABLE; OpenRegionProcedure afccb6ca19c037f46d82ff643517b50b, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:26,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=114 2023-07-12 16:07:26,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:26,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => afccb6ca19c037f46d82ff643517b50b, NAME => 'testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:07:26,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTakeSnapshot afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:26,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:26,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:26,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:26,620 INFO [StoreOpener-afccb6ca19c037f46d82ff643517b50b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:26,622 DEBUG [StoreOpener-afccb6ca19c037f46d82ff643517b50b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1 2023-07-12 16:07:26,622 DEBUG [StoreOpener-afccb6ca19c037f46d82ff643517b50b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1 2023-07-12 16:07:26,622 INFO [StoreOpener-afccb6ca19c037f46d82ff643517b50b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region afccb6ca19c037f46d82ff643517b50b columnFamilyName f1 2023-07-12 16:07:26,623 INFO [StoreOpener-afccb6ca19c037f46d82ff643517b50b-1] regionserver.HStore(310): Store=afccb6ca19c037f46d82ff643517b50b/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:07:26,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:26,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:26,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:26,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:07:26,629 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened afccb6ca19c037f46d82ff643517b50b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74201866, jitterRate=0.10569396615028381}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:07:26,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for afccb6ca19c037f46d82ff643517b50b: 2023-07-12 16:07:26,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b., pid=116, masterSystemTime=1689178046615 2023-07-12 16:07:26,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:26,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:26,631 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=afccb6ca19c037f46d82ff643517b50b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:26,632 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689178046631"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178046631"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178046631"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178046631"}]},"ts":"1689178046631"} 2023-07-12 16:07:26,633 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=115 2023-07-12 16:07:26,634 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=115, state=SUCCESS; OpenRegionProcedure afccb6ca19c037f46d82ff643517b50b, server=jenkins-hbase5.apache.org,38637,1689177958270 in 169 msec 2023-07-12 16:07:26,635 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=114 2023-07-12 16:07:26,635 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=114, state=SUCCESS; TransitRegionStateProcedure table=testTakeSnapshot, region=afccb6ca19c037f46d82ff643517b50b, ASSIGN in 325 msec 2023-07-12 16:07:26,635 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=114, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:07:26,636 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178046636"}]},"ts":"1689178046636"} 2023-07-12 16:07:26,636 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testTakeSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:07:26,640 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=114, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testTakeSnapshot execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:07:26,641 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=114, state=SUCCESS; CreateTableProcedure table=testTakeSnapshot in 1.1640 sec 2023-07-12 16:07:27,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=114 2023-07-12 16:07:27,583 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testTakeSnapshot, procId: 114 completed 2023-07-12 16:07:27,583 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testTakeSnapshot get assigned. Timeout = 60000ms 2023-07-12 16:07:27,583 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:07:27,586 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testTakeSnapshot assigned to meta. Checking AM states. 2023-07-12 16:07:27,586 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:07:27,586 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testTakeSnapshot assigned. 2023-07-12 16:07:27,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38637] ipc.CallRunner(144): callId: 18125 service: ClientService methodName: Mutate size: 143 connection: 172.31.10.131:41838 deadline: 1689178107586, exception=org.apache.hadoop.hbase.NotServingRegionException: testTakeSnapshot,,1689177983276.91762ed77644238ae517369c8db1a1cb. is not online on jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:28,782 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:28,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178048782 (current time:1689178048782). 2023-07-12 16:07:28,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:28,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x2cbe20bd to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:28,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4813a8bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:28,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:28,790 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:36580, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cbe20bd to 127.0.0.1:55421 2023-07-12 16:07:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:28,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=117, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:07:28,794 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=117, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:07:28,794 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=117, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178648794 2023-07-12 16:07:28,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:28,794 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testTakeSnapshot 2023-07-12 16:07:28,795 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=117, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:07:28,795 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:07:28,796 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=117, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE in 2 msec 2023-07-12 16:07:28,796 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:07:28,796 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED; timeout=600000, timestamp=1689178648796 2023-07-12 16:07:28,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:28,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:29,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:29,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:29,207 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:07:29,208 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:07:29,208 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:29,208 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:07:29,208 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:07:29,208 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,208 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:29,210 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:29,210 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:29,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:29,210 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:29,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:29,210 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,210 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 50 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 50 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testTakeSnapshot type FLUSH 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testTakeSnapshot type FLUSH 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:07:29,211 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:07:29,212 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:29,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:07:29,212 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:07:29,212 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:07:29,212 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:29,212 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:07:29,213 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:07:29,213 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:07:29,213 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:07:29,213 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:07:29,214 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,214 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:29,214 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,214 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,214 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:29,215 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,215 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:29,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:07:29,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67204113[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:29,216 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:07:29,216 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,217 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,217 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,217 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:07:29,218 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,218 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,218 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:07:29,218 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:29,218 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:29,218 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:29,218 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:07:29,218 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:29,218 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. started... 2023-07-12 16:07:29,218 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2745): Flushing afccb6ca19c037f46d82ff643517b50b 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:07:29,220 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:07:29,220 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:29,220 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:07:29,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:29,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:29,628 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/.tmp/f1/e76c698e59ca40d1befc0d78e65bba64 2023-07-12 16:07:29,634 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/.tmp/f1/e76c698e59ca40d1befc0d78e65bba64 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1/e76c698e59ca40d1befc0d78e65bba64 2023-07-12 16:07:29,639 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1/e76c698e59ca40d1befc0d78e65bba64, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:07:29,640 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for afccb6ca19c037f46d82ff643517b50b in 422ms, sequenceid=3004, compaction requested=false 2023-07-12 16:07:29,640 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTakeSnapshot' 2023-07-12 16:07:29,640 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for afccb6ca19c037f46d82ff643517b50b: 2023-07-12 16:07:29,640 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.' region-info for snapshot=snapshotName1 2023-07-12 16:07:29,640 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:29,641 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1/e76c698e59ca40d1befc0d78e65bba64] hfiles 2023-07-12 16:07:29,641 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1/e76c698e59ca40d1befc0d78e65bba64 for snapshot=snapshotName1 2023-07-12 16:07:29,645 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. completed. 2023-07-12 16:07:29,645 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:29,646 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:29,646 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:29,646 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,646 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:07:29,646 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:29,647 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,648 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:07:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:29,648 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:29,648 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:07:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:07:29,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:07:29,651 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:07:29,651 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:29,651 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:29,651 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:07:29,651 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:29,653 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,653 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:29,653 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,653 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,653 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:29,653 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:29,654 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:29,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:29,655 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:29,655 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:29,663 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:29,663 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:29,663 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:29,663 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:07:29,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:29,663 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:29,663 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:29,663 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:29,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:29,663 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:29,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/snapshotName1 because node does not exist (not an error) 2023-07-12 16:07:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:29,663 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:07:29,664 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,664 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:29,664 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:29,664 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,664 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:29,664 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:29,664 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:29,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:29,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:29,665 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:29,680 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:07:29,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:29,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:30,097 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testTakeSnapshot completed 2023-07-12 16:07:30,097 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:30,098 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:30,098 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testTakeSnapshot at 1689178048794Consolidate snapshot: snapshotName1 at 1689178049663 (+869 ms)Loading Region manifests for snapshotName1 at 1689178049664 (+1 ms)Writing data manifest for snapshotName1 at 1689178049668 (+4 ms)Verifying snapshot: snapshotName1 at 1689178049673 (+5 ms)Snapshot snapshotName1 of table testTakeSnapshot completed at 1689178050097 (+424 ms) 2023-07-12 16:07:30,100 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:07:30,100 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED in 1.3050 sec 2023-07-12 16:07:30,694 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:07:30,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:30,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testTakeSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:30,840 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:30,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178050840 (current time:1689178050840). 2023-07-12 16:07:30,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:30,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x0ce0d8a0 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:30,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5186ba49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:30,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:30,849 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:36588, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:30,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ce0d8a0 to 127.0.0.1:55421 2023-07-12 16:07:30,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:30,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:30,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:30,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:07:30,853 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=119, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:07:30,853 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=119, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178650853 2023-07-12 16:07:30,853 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testTakeSnapshot 2023-07-12 16:07:30,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:30,855 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=119, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE 2023-07-12 16:07:30,855 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:07:30,855 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=EXCLUSIVE in 3 msec 2023-07-12 16:07:30,856 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=120, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:07:30,856 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=120, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED; timeout=600000, timestamp=1689178650856 2023-07-12 16:07:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:30,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:31,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:31,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:31,264 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName2 2023-07-12 16:07:31,264 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName2' 2023-07-12 16:07:31,264 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:31,264 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName2' starting 'acquire' 2023-07-12 16:07:31,264 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName2', kicking off acquire phase on members. 2023-07-12 16:07:31,265 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,265 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:31,267 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:31,267 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:31,267 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:31,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:31,267 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:31,267 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:31,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 50 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 50 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testTakeSnapshot type FLUSH 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testTakeSnapshot type FLUSH 2023-07-12 16:07:31,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:07:31,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:07:31,269 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:07:31,269 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:31,269 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:07:31,269 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:07:31,269 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:07:31,269 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:07:31,269 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:31,270 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:07:31,270 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:07:31,270 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:07:31,273 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,274 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,274 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,274 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,274 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:31,274 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,274 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:31,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName2' on coordinator 2023-07-12 16:07:31,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8f69cf[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:31,276 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName2' starting 'in-barrier' execution. 2023-07-12 16:07:31,276 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,278 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,278 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,278 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:07:31,278 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:07:31,278 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,278 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:31,278 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:07:31,278 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:31,278 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:31,278 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:31,278 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. started... 2023-07-12 16:07:31,278 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:31,279 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for afccb6ca19c037f46d82ff643517b50b: 2023-07-12 16:07:31,279 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.' region-info for snapshot=snapshotName2 2023-07-12 16:07:31,279 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:31,279 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1/e76c698e59ca40d1befc0d78e65bba64] hfiles 2023-07-12 16:07:31,279 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1/e76c698e59ca40d1befc0d78e65bba64 for snapshot=snapshotName2 2023-07-12 16:07:31,284 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:07:31,284 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:31,284 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:07:31,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:31,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:31,690 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. completed. 2023-07-12 16:07:31,690 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:31,690 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:31,690 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:31,690 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,690 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:07:31,690 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:31,694 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:31,694 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:07:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:31,695 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:31,695 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:07:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName2' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:07:31,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName2', counting down latch. Waiting for 0 more 2023-07-12 16:07:31,698 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName2' execution completed 2023-07-12 16:07:31,698 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:31,698 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:31,698 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName2 2023-07-12 16:07:31,698 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:31,700 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,700 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,700 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:31,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,700 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:31,700 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:31,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:31,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:31,701 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,702 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:31,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,704 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:31,704 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:31,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,707 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:31,707 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,707 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:31,707 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:31,707 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:31,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:31,707 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName2 2023-07-12 16:07:31,707 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,708 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName2 2023-07-12 16:07:31,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:31,707 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:31,707 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:31,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:31,708 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:31,708 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:31,708 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,708 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:31,708 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:31,708 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:31,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:31,709 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:31,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:31,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:31,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:31,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:31,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:31,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:32,125 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName2 2023-07-12 16:07:32,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:32,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:32,944 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName2 of table testTakeSnapshot completed 2023-07-12 16:07:32,944 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:32,944 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:32,944 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testTakeSnapshot at 1689178050854Consolidate snapshot: snapshotName2 at 1689178051708 (+854 ms)Loading Region manifests for snapshotName2 at 1689178051708Writing data manifest for snapshotName2 at 1689178051712 (+4 ms)Verifying snapshot: snapshotName2 at 1689178052118 (+406 ms)Snapshot snapshotName2 of table testTakeSnapshot completed at 1689178052944 (+826 ms) 2023-07-12 16:07:32,946 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=120, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED 2023-07-12 16:07:32,947 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTakeSnapshot, type=SHARED in 2.0920 sec 2023-07-12 16:07:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName2 table=testTakeSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:34,932 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" table: "testTakeSnapshot" creation_time: 1689178048782 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:34,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName2" table: "testTakeSnapshot" creation_time: 1689178050840 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:34,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:07:34,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName2 2023-07-12 16:07:34,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testTakeSnapshot 2023-07-12 16:07:34,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=121, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testTakeSnapshot 2023-07-12 16:07:34,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=121 2023-07-12 16:07:34,938 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178054938"}]},"ts":"1689178054938"} 2023-07-12 16:07:34,939 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testTakeSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:07:34,940 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testTakeSnapshot to state=DISABLING 2023-07-12 16:07:34,941 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=122, ppid=121, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTakeSnapshot, region=afccb6ca19c037f46d82ff643517b50b, UNASSIGN}] 2023-07-12 16:07:34,942 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=121, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTakeSnapshot, region=afccb6ca19c037f46d82ff643517b50b, UNASSIGN 2023-07-12 16:07:34,942 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=afccb6ca19c037f46d82ff643517b50b, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:34,942 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689178054942"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178054942"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178054942"}]},"ts":"1689178054942"} 2023-07-12 16:07:34,943 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=123, ppid=122, state=RUNNABLE; CloseRegionProcedure afccb6ca19c037f46d82ff643517b50b, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:35,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=121 2023-07-12 16:07:35,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:35,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing afccb6ca19c037f46d82ff643517b50b, disabling compactions & flushes 2023-07-12 16:07:35,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:35,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:35,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. after waiting 0 ms 2023-07-12 16:07:35,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:35,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/recovered.edits/3008.seqid, newMaxSeqId=3008, maxSeqId=1 2023-07-12 16:07:35,101 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b. 2023-07-12 16:07:35,101 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for afccb6ca19c037f46d82ff643517b50b: 2023-07-12 16:07:35,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:35,102 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=afccb6ca19c037f46d82ff643517b50b, regionState=CLOSED 2023-07-12 16:07:35,102 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.","families":{"info":[{"qualifier":"regioninfo","vlen":50,"tag":[],"timestamp":"1689178055102"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178055102"}]},"ts":"1689178055102"} 2023-07-12 16:07:35,104 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=123, resume processing ppid=122 2023-07-12 16:07:35,104 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=122, state=SUCCESS; CloseRegionProcedure afccb6ca19c037f46d82ff643517b50b, server=jenkins-hbase5.apache.org,38637,1689177958270 in 160 msec 2023-07-12 16:07:35,104 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=122, resume processing ppid=121 2023-07-12 16:07:35,105 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=121, state=SUCCESS; TransitRegionStateProcedure table=testTakeSnapshot, region=afccb6ca19c037f46d82ff643517b50b, UNASSIGN in 163 msec 2023-07-12 16:07:35,105 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178055105"}]},"ts":"1689178055105"} 2023-07-12 16:07:35,106 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testTakeSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:07:35,107 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testTakeSnapshot to state=DISABLED 2023-07-12 16:07:35,108 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=121, state=SUCCESS; DisableTableProcedure table=testTakeSnapshot in 172 msec 2023-07-12 16:07:35,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=121 2023-07-12 16:07:35,250 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testTakeSnapshot completed 2023-07-12 16:07:35,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testTakeSnapshot 2023-07-12 16:07:35,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:07:35,253 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:07:35,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=124 2023-07-12 16:07:35,253 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=124, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:07:35,257 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:35,259 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/recovered.edits] 2023-07-12 16:07:35,263 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1/e76c698e59ca40d1befc0d78e65bba64 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/f1/e76c698e59ca40d1befc0d78e65bba64 2023-07-12 16:07:35,267 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/recovered.edits/3008.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b/recovered.edits/3008.seqid 2023-07-12 16:07:35,268 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testTakeSnapshot/afccb6ca19c037f46d82ff643517b50b 2023-07-12 16:07:35,268 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testTakeSnapshot regions 2023-07-12 16:07:35,270 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=124, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:07:35,272 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testTakeSnapshot from hbase:meta 2023-07-12 16:07:35,273 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testTakeSnapshot' descriptor. 2023-07-12 16:07:35,274 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=124, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:07:35,274 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testTakeSnapshot' from region states. 2023-07-12 16:07:35,274 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178055274"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:35,275 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:07:35,275 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => afccb6ca19c037f46d82ff643517b50b, NAME => 'testTakeSnapshot,,1689178045475.afccb6ca19c037f46d82ff643517b50b.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:35,275 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testTakeSnapshot' as deleted. 2023-07-12 16:07:35,275 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTakeSnapshot","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689178055275"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:35,277 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testTakeSnapshot state from META 2023-07-12 16:07:35,278 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=124, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testTakeSnapshot 2023-07-12 16:07:35,279 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=124, state=SUCCESS; DeleteTableProcedure table=testTakeSnapshot in 27 msec 2023-07-12 16:07:35,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=124 2023-07-12 16:07:35,359 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testTakeSnapshot 2023-07-12 16:07:35,360 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testTakeSnapshot completed 2023-07-12 16:07:35,373 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testTakeSnapshot[1] Thread=268 (was 271), OpenFileDescriptor=520 (was 518) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=213 (was 181) - SystemLoadAverage LEAK? -, ProcessCount=161 (was 161), AvailableMemoryMB=7031 (was 7044) 2023-07-12 16:07:35,386 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testCloneSnapshot[1] Thread=268, OpenFileDescriptor=520, MaxFileDescriptor=60000, SystemLoadAverage=213, ProcessCount=161, AvailableMemoryMB=7031 2023-07-12 16:07:35,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testCloneSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:07:35,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCloneSnapshot 2023-07-12 16:07:35,389 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:07:35,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testCloneSnapshot" procId is: 125 2023-07-12 16:07:35,390 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:07:35,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=125 2023-07-12 16:07:35,391 DEBUG [HFileArchiver-16] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490 2023-07-12 16:07:35,392 DEBUG [HFileArchiver-16] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490 empty. 2023-07-12 16:07:35,392 DEBUG [HFileArchiver-16] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490 2023-07-12 16:07:35,392 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCloneSnapshot regions 2023-07-12 16:07:35,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=125 2023-07-12 16:07:35,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=125 2023-07-12 16:07:35,803 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/.tabledesc/.tableinfo.0000000001 2023-07-12 16:07:35,804 INFO [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => 33cca7fab0236868a354a5819a019490, NAME => 'testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCloneSnapshot', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:07:35,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=125 2023-07-12 16:07:36,211 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(866): Instantiated testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:36,211 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1604): Closing 33cca7fab0236868a354a5819a019490, disabling compactions & flushes 2023-07-12 16:07:36,211 INFO [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1626): Closing region testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:36,211 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:36,211 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. after waiting 0 ms 2023-07-12 16:07:36,211 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:36,211 INFO [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1838): Closed testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:36,211 DEBUG [RegionOpenAndInit-testCloneSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for 33cca7fab0236868a354a5819a019490: 2023-07-12 16:07:36,213 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:07:36,214 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178056214"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178056214"}]},"ts":"1689178056214"} 2023-07-12 16:07:36,215 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:07:36,216 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:07:36,216 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178056216"}]},"ts":"1689178056216"} 2023-07-12 16:07:36,216 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot, state=ENABLING in hbase:meta 2023-07-12 16:07:36,220 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:07:36,220 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:07:36,221 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:07:36,221 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:07:36,221 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCloneSnapshot, region=33cca7fab0236868a354a5819a019490, ASSIGN}] 2023-07-12 16:07:36,222 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCloneSnapshot, region=33cca7fab0236868a354a5819a019490, ASSIGN 2023-07-12 16:07:36,223 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCloneSnapshot, region=33cca7fab0236868a354a5819a019490, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,38637,1689177958270; forceNewPlan=false, retain=false 2023-07-12 16:07:36,373 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:07:36,374 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=33cca7fab0236868a354a5819a019490, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:36,374 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178056374"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178056374"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178056374"}]},"ts":"1689178056374"} 2023-07-12 16:07:36,375 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure 33cca7fab0236868a354a5819a019490, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:36,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=125 2023-07-12 16:07:36,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:36,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 33cca7fab0236868a354a5819a019490, NAME => 'testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:07:36,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCloneSnapshot 33cca7fab0236868a354a5819a019490 2023-07-12 16:07:36,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:36,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 33cca7fab0236868a354a5819a019490 2023-07-12 16:07:36,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 33cca7fab0236868a354a5819a019490 2023-07-12 16:07:36,532 INFO [StoreOpener-33cca7fab0236868a354a5819a019490-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 33cca7fab0236868a354a5819a019490 2023-07-12 16:07:36,534 DEBUG [StoreOpener-33cca7fab0236868a354a5819a019490-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1 2023-07-12 16:07:36,534 DEBUG [StoreOpener-33cca7fab0236868a354a5819a019490-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1 2023-07-12 16:07:36,534 INFO [StoreOpener-33cca7fab0236868a354a5819a019490-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 33cca7fab0236868a354a5819a019490 columnFamilyName f1 2023-07-12 16:07:36,535 INFO [StoreOpener-33cca7fab0236868a354a5819a019490-1] regionserver.HStore(310): Store=33cca7fab0236868a354a5819a019490/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:07:36,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490 2023-07-12 16:07:36,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490 2023-07-12 16:07:36,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 33cca7fab0236868a354a5819a019490 2023-07-12 16:07:36,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:07:36,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 33cca7fab0236868a354a5819a019490; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59011330, jitterRate=-0.12066265940666199}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:07:36,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 33cca7fab0236868a354a5819a019490: 2023-07-12 16:07:36,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490., pid=127, masterSystemTime=1689178056527 2023-07-12 16:07:36,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:36,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:36,544 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=33cca7fab0236868a354a5819a019490, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:36,545 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178056544"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178056544"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178056544"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178056544"}]},"ts":"1689178056544"} 2023-07-12 16:07:36,546 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-07-12 16:07:36,546 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure 33cca7fab0236868a354a5819a019490, server=jenkins-hbase5.apache.org,38637,1689177958270 in 170 msec 2023-07-12 16:07:36,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=126, resume processing ppid=125 2023-07-12 16:07:36,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=125, state=SUCCESS; TransitRegionStateProcedure table=testCloneSnapshot, region=33cca7fab0236868a354a5819a019490, ASSIGN in 325 msec 2023-07-12 16:07:36,548 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:07:36,548 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178056548"}]},"ts":"1689178056548"} 2023-07-12 16:07:36,549 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot, state=ENABLED in hbase:meta 2023-07-12 16:07:36,551 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCloneSnapshot execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:07:36,552 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=125, state=SUCCESS; CreateTableProcedure table=testCloneSnapshot in 1.1630 sec 2023-07-12 16:07:37,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=125 2023-07-12 16:07:37,495 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCloneSnapshot, procId: 125 completed 2023-07-12 16:07:37,495 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testCloneSnapshot get assigned. Timeout = 60000ms 2023-07-12 16:07:37,496 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:07:37,499 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testCloneSnapshot assigned to meta. Checking AM states. 2023-07-12 16:07:37,499 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:07:37,500 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testCloneSnapshot assigned. 2023-07-12 16:07:37,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.CallRunner(144): callId: 21146 service: ClientService methodName: Mutate size: 144 connection: 172.31.10.131:43266 deadline: 1689178117500, exception=org.apache.hadoop.hbase.NotServingRegionException: testCloneSnapshot,,1689177991823.4c2fefd5ec41ff99dc50f537aa39571a. is not online on jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:38,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:38,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178058704 (current time:1689178058704). 2023-07-12 16:07:38,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:38,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x0a5c91b2 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:38,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44095d0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:38,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:38,711 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:59596, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:38,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a5c91b2 to 127.0.0.1:55421 2023-07-12 16:07:38,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:38,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:38,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:38,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=128, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE 2023-07-12 16:07:38,715 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=128, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE 2023-07-12 16:07:38,716 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=128, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE; timeout=600000, timestamp=1689178658716 2023-07-12 16:07:38,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } 2023-07-12 16:07:38,716 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testCloneSnapshot 2023-07-12 16:07:38,717 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=128, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE 2023-07-12 16:07:38,717 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=129, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED 2023-07-12 16:07:38,717 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=128, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=EXCLUSIVE in 3 msec 2023-07-12 16:07:38,718 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=129, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED 2023-07-12 16:07:38,718 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=129, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED; timeout=600000, timestamp=1689178658718 2023-07-12 16:07:38,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:38,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:38,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:38,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:39,129 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName1 was in running list but was completed. Accepting new attempt. 2023-07-12 16:07:39,129 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:07:39,129 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:07:39,129 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:39,129 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:07:39,129 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:07:39,130 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,130 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:39,131 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:39,131 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:39,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:39,131 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:39,132 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,132 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:39,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:07:39,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:39,133 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testCloneSnapshot type FLUSH 2023-07-12 16:07:39,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:07:39,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:07:39,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,135 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:07:39,135 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:39,136 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:07:39,136 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:07:39,136 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:07:39,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:07:39,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:39,136 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testCloneSnapshot type FLUSH 2023-07-12 16:07:39,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:07:39,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:07:39,136 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:07:39,136 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:39,137 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:07:39,137 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:07:39,137 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:07:39,138 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,138 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:39,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:39,139 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,139 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,139 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:39,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:39,140 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,140 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:39,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:39,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:39,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:39,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:07:39,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a05f1c9[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:39,141 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:07:39,141 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,143 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,143 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:39,143 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:07:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,143 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:07:39,143 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:07:39,143 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:39,143 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:39,143 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:39,143 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:39,143 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. started... 2023-07-12 16:07:39,144 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2745): Flushing 33cca7fab0236868a354a5819a019490 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:07:39,145 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:07:39,145 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:39,145 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:07:39,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:39,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:39,557 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/.tmp/f1/d261fdbf96474776bc1a5d5bc349823f 2023-07-12 16:07:39,564 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/.tmp/f1/d261fdbf96474776bc1a5d5bc349823f as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1/d261fdbf96474776bc1a5d5bc349823f 2023-07-12 16:07:39,568 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1/d261fdbf96474776bc1a5d5bc349823f, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:07:39,568 INFO [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for 33cca7fab0236868a354a5819a019490 in 424ms, sequenceid=3004, compaction requested=false 2023-07-12 16:07:39,569 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 33cca7fab0236868a354a5819a019490: 2023-07-12 16:07:39,569 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.' region-info for snapshot=snapshotName1 2023-07-12 16:07:39,569 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:39,569 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1/d261fdbf96474776bc1a5d5bc349823f] hfiles 2023-07-12 16:07:39,569 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1/d261fdbf96474776bc1a5d5bc349823f for snapshot=snapshotName1 2023-07-12 16:07:39,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:39,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:39,974 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. completed. 2023-07-12 16:07:39,974 DEBUG [rs(jenkins-hbase5.apache.org,38637,1689177958270)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:39,974 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:39,974 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:39,974 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,974 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:07:39,974 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:39,977 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,977 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:07:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:39,977 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:39,977 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:07:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:39,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,38637,1689177958270': 2023-07-12 16:07:39,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:07:39,980 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:07:39,980 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:39,980 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:39,981 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:07:39,981 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:39,985 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,985 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,985 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,985 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,985 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,985 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:39,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:39,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:39,985 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,987 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,987 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,990 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,990 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:39,990 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:39,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:39,990 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:39,990 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:39,990 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:39,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:39,990 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:39,990 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:07:39,990 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,991 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:07:39,991 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:39,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:39,991 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:39,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:39,991 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,991 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:39,991 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:39,991 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:39,992 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:40,406 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:07:40,602 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:07:40,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:40,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:40,821 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testCloneSnapshot completed 2023-07-12 16:07:40,821 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:40,822 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:40,822 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testCloneSnapshot at 1689178058716Consolidate snapshot: snapshotName1 at 1689178059991 (+1275 ms)Loading Region manifests for snapshotName1 at 1689178059991Writing data manifest for snapshotName1 at 1689178059995 (+4 ms)Verifying snapshot: snapshotName1 at 1689178060400 (+405 ms)Snapshot snapshotName1 of table testCloneSnapshot completed at 1689178060821 (+421 ms) 2023-07-12 16:07:40,823 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=129, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED 2023-07-12 16:07:40,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=129, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCloneSnapshot, type=SHARED in 2.1070 sec 2023-07-12 16:07:42,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 } is done 2023-07-12 16:07:42,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testCloneSnapshot type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:42,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=130, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testCloneSnapshot2 snapshot=name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689178058704 type: FLUSH version: 2 ttl: 0 ) 2023-07-12 16:07:42,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(750): Clone snapshot=snapshotName1 as table=testCloneSnapshot2 2023-07-12 16:07:42,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=130 2023-07-12 16:07:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=130 2023-07-12 16:07:43,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=130 2023-07-12 16:07:43,211 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/.tabledesc/.tableinfo.0000000001 2023-07-12 16:07:43,214 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689178058704 type: FLUSH version: 2 ttl: 0 2023-07-12 16:07:43,214 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2 2023-07-12 16:07:43,215 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 33cca7fab0236868a354a5819a019490 2023-07-12 16:07:43,215 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=33cca7fab0236868a354a5819a019490 as 05b37c3ea1b9ac1d0991ad6f54d1a6d3 in snapshot snapshotName1 2023-07-12 16:07:43,216 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => 05b37c3ea1b9ac1d0991ad6f54d1a6d3, NAME => 'testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCloneSnapshot2', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:07:43,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=130 2023-07-12 16:07:43,624 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(866): Instantiated testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:43,624 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d261fdbf96474776bc1a5d5bc349823f from cloned region in snapshot snapshotName1 to table=testCloneSnapshot2 2023-07-12 16:07:43,629 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1604): Closing 05b37c3ea1b9ac1d0991ad6f54d1a6d3, disabling compactions & flushes 2023-07-12 16:07:43,630 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1626): Closing region testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:43,630 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:43,630 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. after waiting 0 ms 2023-07-12 16:07:43,630 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:43,630 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1838): Closed testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:43,630 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for 05b37c3ea1b9ac1d0991ad6f54d1a6d3: 2023-07-12 16:07:43,630 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689178058704 type: FLUSH version: 2 ttl: 0 2023-07-12 16:07:43,630 INFO [PEWorker-3] procedure.CloneSnapshotProcedure$1(421): Clone snapshot=snapshotName1 on table=testCloneSnapshot2 completed! 2023-07-12 16:07:43,634 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689178063634"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178063634"}]},"ts":"1689178063634"} 2023-07-12 16:07:43,635 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:07:43,635 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178063635"}]},"ts":"1689178063635"} 2023-07-12 16:07:43,636 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot2, state=ENABLING in hbase:meta 2023-07-12 16:07:43,640 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:07:43,643 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:07:43,643 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:07:43,643 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:07:43,644 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=130, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCloneSnapshot2, region=05b37c3ea1b9ac1d0991ad6f54d1a6d3, ASSIGN}] 2023-07-12 16:07:43,645 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=131, ppid=130, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCloneSnapshot2, region=05b37c3ea1b9ac1d0991ad6f54d1a6d3, ASSIGN 2023-07-12 16:07:43,645 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=131, ppid=130, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCloneSnapshot2, region=05b37c3ea1b9ac1d0991ad6f54d1a6d3, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,33647,1689177958228; forceNewPlan=false, retain=false 2023-07-12 16:07:43,796 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:07:43,797 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=05b37c3ea1b9ac1d0991ad6f54d1a6d3, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:43,797 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689178063797"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178063797"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178063797"}]},"ts":"1689178063797"} 2023-07-12 16:07:43,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=131, state=RUNNABLE; OpenRegionProcedure 05b37c3ea1b9ac1d0991ad6f54d1a6d3, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:07:43,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=130 2023-07-12 16:07:43,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:43,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 05b37c3ea1b9ac1d0991ad6f54d1a6d3, NAME => 'testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:07:43,954 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCloneSnapshot2 05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:43,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:43,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:43,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:43,956 INFO [StoreOpener-05b37c3ea1b9ac1d0991ad6f54d1a6d3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:43,957 DEBUG [StoreOpener-05b37c3ea1b9ac1d0991ad6f54d1a6d3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/f1 2023-07-12 16:07:43,957 DEBUG [StoreOpener-05b37c3ea1b9ac1d0991ad6f54d1a6d3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/f1 2023-07-12 16:07:43,957 INFO [StoreOpener-05b37c3ea1b9ac1d0991ad6f54d1a6d3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 05b37c3ea1b9ac1d0991ad6f54d1a6d3 columnFamilyName f1 2023-07-12 16:07:43,962 DEBUG [StoreOpener-05b37c3ea1b9ac1d0991ad6f54d1a6d3-1] regionserver.HStore(539): loaded hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/f1/testCloneSnapshot=33cca7fab0236868a354a5819a019490-d261fdbf96474776bc1a5d5bc349823f 2023-07-12 16:07:43,963 INFO [StoreOpener-05b37c3ea1b9ac1d0991ad6f54d1a6d3-1] regionserver.HStore(310): Store=05b37c3ea1b9ac1d0991ad6f54d1a6d3/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:07:43,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:43,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:43,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:43,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/recovered.edits/3004.seqid, newMaxSeqId=3004, maxSeqId=-1 2023-07-12 16:07:43,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 05b37c3ea1b9ac1d0991ad6f54d1a6d3; next sequenceid=3005; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63478558, jitterRate=-0.05409577488899231}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:07:43,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 05b37c3ea1b9ac1d0991ad6f54d1a6d3: 2023-07-12 16:07:43,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3., pid=132, masterSystemTime=1689178063949 2023-07-12 16:07:43,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:43,970 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:43,970 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=05b37c3ea1b9ac1d0991ad6f54d1a6d3, regionState=OPEN, openSeqNum=3005, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:43,971 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689178063970"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178063970"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178063970"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178063970"}]},"ts":"1689178063970"} 2023-07-12 16:07:43,972 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=131 2023-07-12 16:07:43,972 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=131, state=SUCCESS; OpenRegionProcedure 05b37c3ea1b9ac1d0991ad6f54d1a6d3, server=jenkins-hbase5.apache.org,33647,1689177958228 in 173 msec 2023-07-12 16:07:43,973 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=130 2023-07-12 16:07:43,973 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=130, state=SUCCESS; TransitRegionStateProcedure table=testCloneSnapshot2, region=05b37c3ea1b9ac1d0991ad6f54d1a6d3, ASSIGN in 328 msec 2023-07-12 16:07:43,973 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178063973"}]},"ts":"1689178063973"} 2023-07-12 16:07:43,974 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot2, state=ENABLED in hbase:meta 2023-07-12 16:07:43,977 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=130, state=SUCCESS; CloneSnapshotProcedure (table=testCloneSnapshot2 snapshot=name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689178058704 type: FLUSH version: 2 ttl: 0 ) in 1.1750 sec 2023-07-12 16:07:44,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=130 2023-07-12 16:07:44,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" table: "testCloneSnapshot" creation_time: 1689178058704 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:44,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:07:44,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testCloneSnapshot 2023-07-12 16:07:44,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCloneSnapshot 2023-07-12 16:07:44,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=133 2023-07-12 16:07:44,962 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178064961"}]},"ts":"1689178064961"} 2023-07-12 16:07:44,962 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot, state=DISABLING in hbase:meta 2023-07-12 16:07:44,964 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testCloneSnapshot to state=DISABLING 2023-07-12 16:07:44,964 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCloneSnapshot, region=33cca7fab0236868a354a5819a019490, UNASSIGN}] 2023-07-12 16:07:44,965 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCloneSnapshot, region=33cca7fab0236868a354a5819a019490, UNASSIGN 2023-07-12 16:07:44,965 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=33cca7fab0236868a354a5819a019490, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:44,966 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178064965"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178064965"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178064965"}]},"ts":"1689178064965"} 2023-07-12 16:07:44,966 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=135, ppid=134, state=RUNNABLE; CloseRegionProcedure 33cca7fab0236868a354a5819a019490, server=jenkins-hbase5.apache.org,38637,1689177958270}] 2023-07-12 16:07:45,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=133 2023-07-12 16:07:45,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 33cca7fab0236868a354a5819a019490 2023-07-12 16:07:45,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 33cca7fab0236868a354a5819a019490, disabling compactions & flushes 2023-07-12 16:07:45,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:45,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:45,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. after waiting 0 ms 2023-07-12 16:07:45,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:45,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/recovered.edits/3007.seqid, newMaxSeqId=3007, maxSeqId=1 2023-07-12 16:07:45,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490. 2023-07-12 16:07:45,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 33cca7fab0236868a354a5819a019490: 2023-07-12 16:07:45,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 33cca7fab0236868a354a5819a019490 2023-07-12 16:07:45,125 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=33cca7fab0236868a354a5819a019490, regionState=CLOSED 2023-07-12 16:07:45,125 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178065125"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178065125"}]},"ts":"1689178065125"} 2023-07-12 16:07:45,126 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=135, resume processing ppid=134 2023-07-12 16:07:45,126 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=134, state=SUCCESS; CloseRegionProcedure 33cca7fab0236868a354a5819a019490, server=jenkins-hbase5.apache.org,38637,1689177958270 in 160 msec 2023-07-12 16:07:45,127 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=134, resume processing ppid=133 2023-07-12 16:07:45,127 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testCloneSnapshot, region=33cca7fab0236868a354a5819a019490, UNASSIGN in 162 msec 2023-07-12 16:07:45,128 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178065128"}]},"ts":"1689178065128"} 2023-07-12 16:07:45,128 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot, state=DISABLED in hbase:meta 2023-07-12 16:07:45,130 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCloneSnapshot to state=DISABLED 2023-07-12 16:07:45,130 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testCloneSnapshot in 171 msec 2023-07-12 16:07:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=133 2023-07-12 16:07:45,280 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testCloneSnapshot completed 2023-07-12 16:07:45,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testCloneSnapshot2 2023-07-12 16:07:45,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=136, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCloneSnapshot2 2023-07-12 16:07:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=136 2023-07-12 16:07:45,284 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178065284"}]},"ts":"1689178065284"} 2023-07-12 16:07:45,284 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot2, state=DISABLING in hbase:meta 2023-07-12 16:07:45,287 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCloneSnapshot2 to state=DISABLING 2023-07-12 16:07:45,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=137, ppid=136, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCloneSnapshot2, region=05b37c3ea1b9ac1d0991ad6f54d1a6d3, UNASSIGN}] 2023-07-12 16:07:45,289 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=136, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCloneSnapshot2, region=05b37c3ea1b9ac1d0991ad6f54d1a6d3, UNASSIGN 2023-07-12 16:07:45,289 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=05b37c3ea1b9ac1d0991ad6f54d1a6d3, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:45,290 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689178065289"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178065289"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178065289"}]},"ts":"1689178065289"} 2023-07-12 16:07:45,291 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=138, ppid=137, state=RUNNABLE; CloseRegionProcedure 05b37c3ea1b9ac1d0991ad6f54d1a6d3, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:07:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=136 2023-07-12 16:07:45,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:45,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 05b37c3ea1b9ac1d0991ad6f54d1a6d3, disabling compactions & flushes 2023-07-12 16:07:45,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:45,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:45,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. after waiting 0 ms 2023-07-12 16:07:45,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:45,454 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/recovered.edits/3007.seqid, newMaxSeqId=3007, maxSeqId=3004 2023-07-12 16:07:45,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3. 2023-07-12 16:07:45,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 05b37c3ea1b9ac1d0991ad6f54d1a6d3: 2023-07-12 16:07:45,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:45,457 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=05b37c3ea1b9ac1d0991ad6f54d1a6d3, regionState=CLOSED 2023-07-12 16:07:45,457 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689178065457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178065457"}]},"ts":"1689178065457"} 2023-07-12 16:07:45,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=138, resume processing ppid=137 2023-07-12 16:07:45,460 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=137, state=SUCCESS; CloseRegionProcedure 05b37c3ea1b9ac1d0991ad6f54d1a6d3, server=jenkins-hbase5.apache.org,33647,1689177958228 in 167 msec 2023-07-12 16:07:45,461 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=137, resume processing ppid=136 2023-07-12 16:07:45,461 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=136, state=SUCCESS; TransitRegionStateProcedure table=testCloneSnapshot2, region=05b37c3ea1b9ac1d0991ad6f54d1a6d3, UNASSIGN in 172 msec 2023-07-12 16:07:45,461 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178065461"}]},"ts":"1689178065461"} 2023-07-12 16:07:45,462 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCloneSnapshot2, state=DISABLED in hbase:meta 2023-07-12 16:07:45,464 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCloneSnapshot2 to state=DISABLED 2023-07-12 16:07:45,466 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=136, state=SUCCESS; DisableTableProcedure table=testCloneSnapshot2 in 184 msec 2023-07-12 16:07:45,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=136 2023-07-12 16:07:45,599 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testCloneSnapshot2 completed 2023-07-12 16:07:45,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testCloneSnapshot 2023-07-12 16:07:45,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=139, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:07:45,602 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=139, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:07:45,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=139 2023-07-12 16:07:45,602 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=139, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:07:45,606 DEBUG [HFileArchiver-17] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490 2023-07-12 16:07:45,607 DEBUG [HFileArchiver-17] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/recovered.edits] 2023-07-12 16:07:45,611 DEBUG [HFileArchiver-17] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1/d261fdbf96474776bc1a5d5bc349823f to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/f1/d261fdbf96474776bc1a5d5bc349823f 2023-07-12 16:07:45,614 DEBUG [HFileArchiver-17] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/recovered.edits/3007.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490/recovered.edits/3007.seqid 2023-07-12 16:07:45,615 DEBUG [HFileArchiver-17] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot/33cca7fab0236868a354a5819a019490 2023-07-12 16:07:45,615 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCloneSnapshot regions 2023-07-12 16:07:45,617 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=139, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:07:45,618 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testCloneSnapshot from hbase:meta 2023-07-12 16:07:45,620 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testCloneSnapshot' descriptor. 2023-07-12 16:07:45,620 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=139, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:07:45,620 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testCloneSnapshot' from region states. 2023-07-12 16:07:45,620 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178065620"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:45,622 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:07:45,622 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 33cca7fab0236868a354a5819a019490, NAME => 'testCloneSnapshot,,1689178055388.33cca7fab0236868a354a5819a019490.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:45,622 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testCloneSnapshot' as deleted. 2023-07-12 16:07:45,622 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCloneSnapshot","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689178065622"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:45,623 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testCloneSnapshot state from META 2023-07-12 16:07:45,625 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=139, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCloneSnapshot 2023-07-12 16:07:45,625 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=139, state=SUCCESS; DeleteTableProcedure table=testCloneSnapshot in 25 msec 2023-07-12 16:07:45,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=139 2023-07-12 16:07:45,710 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testCloneSnapshot 2023-07-12 16:07:45,710 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testCloneSnapshot completed 2023-07-12 16:07:45,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testCloneSnapshot2 2023-07-12 16:07:45,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:07:45,712 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:07:45,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=140 2023-07-12 16:07:45,713 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=140, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:07:45,716 DEBUG [HFileArchiver-18] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:45,718 DEBUG [HFileArchiver-18] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/recovered.edits] 2023-07-12 16:07:45,721 DEBUG [HFileArchiver-18] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/f1/testCloneSnapshot=33cca7fab0236868a354a5819a019490-d261fdbf96474776bc1a5d5bc349823f to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/f1/testCloneSnapshot=33cca7fab0236868a354a5819a019490-d261fdbf96474776bc1a5d5bc349823f 2023-07-12 16:07:45,724 DEBUG [HFileArchiver-18] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/recovered.edits/3007.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3/recovered.edits/3007.seqid 2023-07-12 16:07:45,725 DEBUG [HFileArchiver-18] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testCloneSnapshot2/05b37c3ea1b9ac1d0991ad6f54d1a6d3 2023-07-12 16:07:45,725 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testCloneSnapshot2 regions 2023-07-12 16:07:45,727 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=140, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:07:45,728 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testCloneSnapshot2 from hbase:meta 2023-07-12 16:07:45,730 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testCloneSnapshot2' descriptor. 2023-07-12 16:07:45,730 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=140, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:07:45,730 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testCloneSnapshot2' from region states. 2023-07-12 16:07:45,731 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178065730"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:45,731 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:07:45,732 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 05b37c3ea1b9ac1d0991ad6f54d1a6d3, NAME => 'testCloneSnapshot2,,1689178055388.05b37c3ea1b9ac1d0991ad6f54d1a6d3.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:45,732 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testCloneSnapshot2' as deleted. 2023-07-12 16:07:45,732 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCloneSnapshot2","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689178065732"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:45,733 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testCloneSnapshot2 state from META 2023-07-12 16:07:45,735 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=140, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCloneSnapshot2 2023-07-12 16:07:45,735 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=140, state=SUCCESS; DeleteTableProcedure table=testCloneSnapshot2 in 24 msec 2023-07-12 16:07:45,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=140 2023-07-12 16:07:45,820 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testCloneSnapshot2 2023-07-12 16:07:45,820 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testCloneSnapshot2 completed 2023-07-12 16:07:45,834 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testCloneSnapshot[1] Thread=271 (was 268) - Thread LEAK? -, OpenFileDescriptor=517 (was 520), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=209 (was 213), ProcessCount=161 (was 161), AvailableMemoryMB=6747 (was 7031) 2023-07-12 16:07:45,845 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testDeleteSnapshots[1] Thread=271, OpenFileDescriptor=517, MaxFileDescriptor=60000, SystemLoadAverage=209, ProcessCount=161, AvailableMemoryMB=6746 2023-07-12 16:07:45,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testDeleteSnapshots', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:07:45,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testDeleteSnapshots 2023-07-12 16:07:45,849 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:07:45,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testDeleteSnapshots" procId is: 141 2023-07-12 16:07:45,849 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:07:45,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=141 2023-07-12 16:07:45,851 DEBUG [HFileArchiver-19] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:45,851 DEBUG [HFileArchiver-19] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1 empty. 2023-07-12 16:07:45,852 DEBUG [HFileArchiver-19] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:45,852 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testDeleteSnapshots regions 2023-07-12 16:07:45,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=141 2023-07-12 16:07:46,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=141 2023-07-12 16:07:46,259 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/.tabledesc/.tableinfo.0000000001 2023-07-12 16:07:46,261 INFO [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(7675): creating {ENCODED => 40bb34095ed73242a70fbe098defe2a1, NAME => 'testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testDeleteSnapshots', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:07:46,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=141 2023-07-12 16:07:46,667 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(866): Instantiated testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:46,667 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1604): Closing 40bb34095ed73242a70fbe098defe2a1, disabling compactions & flushes 2023-07-12 16:07:46,667 INFO [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1626): Closing region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:46,667 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:46,667 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1714): Acquired close lock on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. after waiting 0 ms 2023-07-12 16:07:46,667 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1724): Updates disabled for region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:46,667 INFO [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1838): Closed testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:46,667 DEBUG [RegionOpenAndInit-testDeleteSnapshots-pool-0] regionserver.HRegion(1558): Region close journal for 40bb34095ed73242a70fbe098defe2a1: 2023-07-12 16:07:46,669 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:07:46,670 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178066670"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178066670"}]},"ts":"1689178066670"} 2023-07-12 16:07:46,671 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:07:46,672 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:07:46,672 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178066672"}]},"ts":"1689178066672"} 2023-07-12 16:07:46,673 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteSnapshots, state=ENABLING in hbase:meta 2023-07-12 16:07:46,676 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:07:46,677 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:07:46,677 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:07:46,677 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:07:46,677 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteSnapshots, region=40bb34095ed73242a70fbe098defe2a1, ASSIGN}] 2023-07-12 16:07:46,679 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteSnapshots, region=40bb34095ed73242a70fbe098defe2a1, ASSIGN 2023-07-12 16:07:46,679 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDeleteSnapshots, region=40bb34095ed73242a70fbe098defe2a1, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,33647,1689177958228; forceNewPlan=false, retain=false 2023-07-12 16:07:46,829 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:07:46,830 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=40bb34095ed73242a70fbe098defe2a1, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:46,831 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178066830"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178066830"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178066830"}]},"ts":"1689178066830"} 2023-07-12 16:07:46,832 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=142, state=RUNNABLE; OpenRegionProcedure 40bb34095ed73242a70fbe098defe2a1, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:07:46,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=141 2023-07-12 16:07:46,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:46,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 40bb34095ed73242a70fbe098defe2a1, NAME => 'testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:07:46,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDeleteSnapshots 40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:46,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:07:46,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:46,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:46,987 INFO [StoreOpener-40bb34095ed73242a70fbe098defe2a1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:46,989 DEBUG [StoreOpener-40bb34095ed73242a70fbe098defe2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1 2023-07-12 16:07:46,989 DEBUG [StoreOpener-40bb34095ed73242a70fbe098defe2a1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1 2023-07-12 16:07:46,989 INFO [StoreOpener-40bb34095ed73242a70fbe098defe2a1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 40bb34095ed73242a70fbe098defe2a1 columnFamilyName f1 2023-07-12 16:07:46,990 INFO [StoreOpener-40bb34095ed73242a70fbe098defe2a1-1] regionserver.HStore(310): Store=40bb34095ed73242a70fbe098defe2a1/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:07:46,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:46,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:46,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:46,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:07:46,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 40bb34095ed73242a70fbe098defe2a1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67545040, jitterRate=0.006499528884887695}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:07:46,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 40bb34095ed73242a70fbe098defe2a1: 2023-07-12 16:07:46,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1., pid=143, masterSystemTime=1689178066982 2023-07-12 16:07:47,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:47,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:47,000 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=40bb34095ed73242a70fbe098defe2a1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:47,001 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178067000"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178067000"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178067000"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178067000"}]},"ts":"1689178067000"} 2023-07-12 16:07:47,002 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=142 2023-07-12 16:07:47,002 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=142, state=SUCCESS; OpenRegionProcedure 40bb34095ed73242a70fbe098defe2a1, server=jenkins-hbase5.apache.org,33647,1689177958228 in 170 msec 2023-07-12 16:07:47,003 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=141 2023-07-12 16:07:47,004 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=141, state=SUCCESS; TransitRegionStateProcedure table=testDeleteSnapshots, region=40bb34095ed73242a70fbe098defe2a1, ASSIGN in 325 msec 2023-07-12 16:07:47,004 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:07:47,004 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178067004"}]},"ts":"1689178067004"} 2023-07-12 16:07:47,005 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteSnapshots, state=ENABLED in hbase:meta 2023-07-12 16:07:47,007 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testDeleteSnapshots execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:07:47,008 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=141, state=SUCCESS; CreateTableProcedure table=testDeleteSnapshots in 1.1600 sec 2023-07-12 16:07:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=141 2023-07-12 16:07:47,954 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testDeleteSnapshots, procId: 141 completed 2023-07-12 16:07:47,954 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testDeleteSnapshots get assigned. Timeout = 60000ms 2023-07-12 16:07:47,954 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:07:47,957 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testDeleteSnapshots assigned to meta. Checking AM states. 2023-07-12 16:07:47,957 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:07:47,957 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testDeleteSnapshots assigned. 2023-07-12 16:07:47,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38637] ipc.CallRunner(144): callId: 24169 service: ClientService methodName: Mutate size: 146 connection: 172.31.10.131:41838 deadline: 1689178127957, exception=org.apache.hadoop.hbase.NotServingRegionException: testDeleteSnapshots,,1689177996119.8f86f373a8ffe82c020499e3ca1d249e. is not online on jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:49,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:49,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178069091 (current time:1689178069091). 2023-07-12 16:07:49,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:49,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x0d63a669 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:49,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25d7c36, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:49,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:49,100 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:49,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d63a669 to 127.0.0.1:55421 2023-07-12 16:07:49,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:49,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:49,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=144, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:49,104 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=144, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:49,105 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=144, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178669105 2023-07-12 16:07:49,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:49,105 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots 2023-07-12 16:07:49,106 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=144, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:49,106 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=145, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:49,106 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE in 3 msec 2023-07-12 16:07:49,107 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=145, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:49,107 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=145, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED; timeout=600000, timestamp=1689178669107 2023-07-12 16:07:49,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:49,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:49,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:49,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:49,456 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:07:49,486 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testDeleteSnapshots' 2023-07-12 16:07:49,514 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName1 was in running list but was completed. Accepting new attempt. 2023-07-12 16:07:49,514 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:07:49,514 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:07:49,514 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:49,514 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:07:49,514 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:07:49,515 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:49,515 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:49,516 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:49,516 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:49,516 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:49,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:49,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:49,516 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:49,517 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:49,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:49,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:49,517 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testDeleteSnapshots type FLUSH 2023-07-12 16:07:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:49,518 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testDeleteSnapshots type FLUSH 2023-07-12 16:07:49,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:07:49,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:07:49,518 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:07:49,518 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:49,519 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:07:49,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:07:49,519 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:07:49,519 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:07:49,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:07:49,519 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:07:49,519 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:49,520 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:07:49,520 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:07:49,520 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:07:49,521 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,522 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:49,522 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:49,522 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,522 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:49,523 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,523 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:49,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:07:49,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c5ca0a0[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:49,524 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:07:49,525 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,527 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,527 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:49,527 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:07:49,527 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:07:49,527 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:07:49,527 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:49,527 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:49,527 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:49,527 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:49,527 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:49,527 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:49,527 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. started... 2023-07-12 16:07:49,528 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2745): Flushing 40bb34095ed73242a70fbe098defe2a1 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:07:49,529 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:07:49,529 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:49,529 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:07:49,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:49,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:49,944 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/.tmp/f1/975a8aa4c4994aa1aec6a8e3d07e9658 2023-07-12 16:07:49,967 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/.tmp/f1/975a8aa4c4994aa1aec6a8e3d07e9658 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658 2023-07-12 16:07:49,975 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:07:49,975 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for 40bb34095ed73242a70fbe098defe2a1 in 447ms, sequenceid=3004, compaction requested=false 2023-07-12 16:07:49,976 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 40bb34095ed73242a70fbe098defe2a1: 2023-07-12 16:07:49,977 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.' region-info for snapshot=snapshotName1 2023-07-12 16:07:49,977 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:49,977 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658] hfiles 2023-07-12 16:07:49,977 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658 for snapshot=snapshotName1 2023-07-12 16:07:49,993 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. completed. 2023-07-12 16:07:49,993 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:49,993 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:49,994 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:49,994 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:49,994 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:07:49,994 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:50,001 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,001 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:07:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:50,001 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:50,001 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:07:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:07:50,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:07:50,006 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:07:50,006 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:50,006 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:50,006 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:07:50,006 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:50,008 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,008 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,008 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:50,008 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:50,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,008 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:50,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,013 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:50,013 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:07:50,015 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:50,016 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,019 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,019 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:50,019 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:50,019 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:50,019 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:50,020 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:50,020 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:50,020 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:50,020 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:07:50,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:50,020 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:07:50,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:50,021 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:50,021 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:07:50,021 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:50,021 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:50,021 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:07:50,021 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:07:50,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:50,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:50,022 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:50,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:50,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:50,441 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:07:51,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:51,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:51,256 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testDeleteSnapshots completed 2023-07-12 16:07:51,256 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:51,257 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:07:51,257 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots at 1689178069105Consolidate snapshot: snapshotName1 at 1689178070020 (+915 ms)Loading Region manifests for snapshotName1 at 1689178070021 (+1 ms)Writing data manifest for snapshotName1 at 1689178070025 (+4 ms)Verifying snapshot: snapshotName1 at 1689178070434 (+409 ms)Snapshot snapshotName1 of table testDeleteSnapshots completed at 1689178071256 (+822 ms) 2023-07-12 16:07:51,258 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=145, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:51,259 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=145, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED in 2.1530 sec 2023-07-12 16:07:53,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testDeleteSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:53,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178073170 (current time:1689178073170). 2023-07-12 16:07:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x38d57cd6 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:53,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2895c54b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:53,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:53,182 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34870, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:53,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38d57cd6 to 127.0.0.1:55421 2023-07-12 16:07:53,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:53,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:53,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:53,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:53,188 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:53,189 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178673189 2023-07-12 16:07:53,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:53,189 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots 2023-07-12 16:07:53,191 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:53,191 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:53,192 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE in 4 msec 2023-07-12 16:07:53,192 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=147, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:53,193 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=147, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED; timeout=600000, timestamp=1689178673193 2023-07-12 16:07:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:53,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:53,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:53,602 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName2 was in running list but was completed. Accepting new attempt. 2023-07-12 16:07:53,602 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName2 2023-07-12 16:07:53,602 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName2' 2023-07-12 16:07:53,602 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:53,603 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName2' starting 'acquire' 2023-07-12 16:07:53,603 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName2', kicking off acquire phase on members. 2023-07-12 16:07:53,603 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:53,603 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:53,605 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:53,605 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:53,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:53,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:53,605 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:53,606 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:53,606 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:53,606 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testDeleteSnapshots type FLUSH 2023-07-12 16:07:53,607 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testDeleteSnapshots type FLUSH 2023-07-12 16:07:53,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:07:53,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:07:53,607 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:07:53,607 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:53,608 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:07:53,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:07:53,608 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:07:53,608 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:07:53,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:07:53,608 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:07:53,608 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:53,608 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:07:53,608 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:07:53,608 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:07:53,609 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,610 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,610 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:53,610 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,610 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:53,610 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,610 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:53,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName2' on coordinator 2023-07-12 16:07:53,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@291148e1[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:53,612 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName2' starting 'in-barrier' execution. 2023-07-12 16:07:53,612 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,613 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,613 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:53,613 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:07:53,614 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:07:53,614 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:07:53,613 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:53,614 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:53,614 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:53,614 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:53,614 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. started... 2023-07-12 16:07:53,614 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:53,614 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:53,614 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 40bb34095ed73242a70fbe098defe2a1: 2023-07-12 16:07:53,614 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.' region-info for snapshot=snapshotName2 2023-07-12 16:07:53,614 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:53,614 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658] hfiles 2023-07-12 16:07:53,614 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658 for snapshot=snapshotName2 2023-07-12 16:07:53,616 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:07:53,616 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:53,616 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:07:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:54,020 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. completed. 2023-07-12 16:07:54,020 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:54,020 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:54,021 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:54,021 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,021 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:07:54,021 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:54,022 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,022 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:07:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:54,022 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:54,023 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:07:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName2' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:07:54,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName2', counting down latch. Waiting for 0 more 2023-07-12 16:07:54,025 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName2' execution completed 2023-07-12 16:07:54,026 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:54,026 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:54,026 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName2 2023-07-12 16:07:54,026 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:54,027 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,027 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,027 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:54,027 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,027 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,028 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:54,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:54,028 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,029 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:54,030 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:07:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,032 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,032 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:54,032 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:54,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:54,032 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:54,032 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:54,032 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName2 2023-07-12 16:07:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:54,032 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:54,032 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:54,032 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName2 2023-07-12 16:07:54,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:54,032 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:54,032 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:07:54,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:54,033 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:54,033 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:54,033 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:54,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:54,033 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:07:54,033 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:07:54,034 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:54,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:54,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:54,448 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName2 2023-07-12 16:07:54,471 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName2 of table testDeleteSnapshots completed 2023-07-12 16:07:54,471 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:54,472 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:07:54,472 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots at 1689178073189Consolidate snapshot: snapshotName2 at 1689178074032 (+843 ms)Loading Region manifests for snapshotName2 at 1689178074033 (+1 ms)Writing data manifest for snapshotName2 at 1689178074036 (+3 ms)Verifying snapshot: snapshotName2 at 1689178074442 (+406 ms)Snapshot snapshotName2 of table testDeleteSnapshots completed at 1689178074471 (+29 ms) 2023-07-12 16:07:54,473 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=147, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:54,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED in 1.2830 sec 2023-07-12 16:07:55,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName2 table=testDeleteSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:55,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178075240 (current time:1689178075240). 2023-07-12 16:07:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:07:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x0bdf3c67 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:07:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@187fc206, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:07:55,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:07:55,248 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34872, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:07:55,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bdf3c67 to 127.0.0.1:55421 2023-07-12 16:07:55,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:07:55,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:07:55,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:55,252 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=148, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:55,253 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=148, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178675253 2023-07-12 16:07:55,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } 2023-07-12 16:07:55,253 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName3 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots 2023-07-12 16:07:55,254 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=148, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE 2023-07-12 16:07:55,254 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=149, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:55,254 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=EXCLUSIVE in 3 msec 2023-07-12 16:07:55,255 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=149, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:55,255 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=149, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED; timeout=600000, timestamp=1689178675255 2023-07-12 16:07:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:55,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:55,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:55,663 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName3 2023-07-12 16:07:55,663 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName3' 2023-07-12 16:07:55,663 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:55,663 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName3' starting 'acquire' 2023-07-12 16:07:55,663 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName3', kicking off acquire phase on members. 2023-07-12 16:07:55,664 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:55,664 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:55,666 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:55,666 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:55,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:55,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:55,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:55,667 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:55,667 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:55,667 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName3 from table testDeleteSnapshots type FLUSH 2023-07-12 16:07:55,667 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:07:55,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 53 2023-07-12 16:07:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:55,668 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName3 from table testDeleteSnapshots type FLUSH 2023-07-12 16:07:55,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName3 2023-07-12 16:07:55,671 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName3' with timeout 300000ms 2023-07-12 16:07:55,671 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:55,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName3 2023-07-12 16:07:55,679 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName3' with timeout 300000ms 2023-07-12 16:07:55,680 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName3' starting 'acquire' stage 2023-07-12 16:07:55,680 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName3' locally acquired 2023-07-12 16:07:55,680 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:07:55,680 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName3) in zk 2023-07-12 16:07:55,681 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName3' starting 'acquire' stage 2023-07-12 16:07:55,681 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName3' locally acquired 2023-07-12 16:07:55,681 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName3) in zk 2023-07-12 16:07:55,683 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,683 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:55,683 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:55,683 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,683 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:55,684 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,685 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:07:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:55,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName3' on coordinator 2023-07-12 16:07:55,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da48d1d[Count = 0] remaining members to acquire global barrier 2023-07-12 16:07:55,688 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName3' starting 'in-barrier' execution. 2023-07-12 16:07:55,688 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,690 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,690 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:55,691 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName3' received 'reached' from coordinator. 2023-07-12 16:07:55,691 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName3' received 'reached' from coordinator. 2023-07-12 16:07:55,691 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName3' locally completed 2023-07-12 16:07:55,691 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName3' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:07:55,691 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:55,691 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:07:55,691 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:07:55,691 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:07:55,691 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:55,691 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. started... 2023-07-12 16:07:55,691 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 40bb34095ed73242a70fbe098defe2a1: 2023-07-12 16:07:55,692 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.' region-info for snapshot=snapshotName3 2023-07-12 16:07:55,692 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:07:55,692 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658] hfiles 2023-07-12 16:07:55,692 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658 for snapshot=snapshotName3 2023-07-12 16:07:55,697 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName3' has notified controller of completion 2023-07-12 16:07:55,697 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:55,697 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName3' completed. 2023-07-12 16:07:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:56,105 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. completed. 2023-07-12 16:07:56,105 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:56,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:07:56,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:07:56,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName3' locally completed 2023-07-12 16:07:56,106 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName3' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:07:56,107 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,107 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName3' has notified controller of completion 2023-07-12 16:07:56,108 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:56,108 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName3' completed. 2023-07-12 16:07:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName3' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:07:56,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName3', counting down latch. Waiting for 0 more 2023-07-12 16:07:56,113 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName3' execution completed 2023-07-12 16:07:56,113 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:07:56,113 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:07:56,113 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName3 2023-07-12 16:07:56,114 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName3including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:07:56,115 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,116 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,116 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:56,115 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,116 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:56,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:56,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:07:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:07:56,117 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:07:56,118 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:07:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,120 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:07:56,121 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:07:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,125 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:56,126 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:07:56,126 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:56,126 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:56,126 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:07:56,126 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName3 2023-07-12 16:07:56,126 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:56,127 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName3 2023-07-12 16:07:56,126 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:56,126 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:07:56,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:07:56,127 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:07:56,127 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:56,127 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:56,127 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:07:56,127 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:07:56,128 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:07:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:07:56,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:56,128 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:07:56,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:07:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:07:56,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:56,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:56,575 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName3 2023-07-12 16:07:57,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:57,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:07:57,391 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName3 of table testDeleteSnapshots completed 2023-07-12 16:07:57,391 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:07:57,391 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:07:57,391 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName3 C_M_SNAPSHOT_TABLE on table testDeleteSnapshots at 1689178075253Consolidate snapshot: snapshotName3 at 1689178076127 (+874 ms)Loading Region manifests for snapshotName3 at 1689178076127Writing data manifest for snapshotName3 at 1689178076149 (+22 ms)Verifying snapshot: snapshotName3 at 1689178076569 (+420 ms)Snapshot snapshotName3 of table testDeleteSnapshots completed at 1689178077391 (+822 ms) 2023-07-12 16:07:57,393 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=149, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED 2023-07-12 16:07:57,393 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=149, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testDeleteSnapshots, type=SHARED in 2.1400 sec 2023-07-12 16:07:59,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:07:59,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName3 table=testDeleteSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:07:59,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" type: DISABLED 2023-07-12 16:07:59,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:07:59,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName2" table: "testDeleteSnapshots" creation_time: 1689178073170 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName2 2023-07-12 16:07:59,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName3" table: "testDeleteSnapshots" creation_time: 1689178075240 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:07:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName3 2023-07-12 16:07:59,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testDeleteSnapshots 2023-07-12 16:07:59,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=150, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDeleteSnapshots 2023-07-12 16:07:59,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=150 2023-07-12 16:07:59,390 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178079390"}]},"ts":"1689178079390"} 2023-07-12 16:07:59,390 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteSnapshots, state=DISABLING in hbase:meta 2023-07-12 16:07:59,392 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDeleteSnapshots to state=DISABLING 2023-07-12 16:07:59,392 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=151, ppid=150, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteSnapshots, region=40bb34095ed73242a70fbe098defe2a1, UNASSIGN}] 2023-07-12 16:07:59,393 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=150, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteSnapshots, region=40bb34095ed73242a70fbe098defe2a1, UNASSIGN 2023-07-12 16:07:59,394 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=40bb34095ed73242a70fbe098defe2a1, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:07:59,394 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178079393"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178079393"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178079393"}]},"ts":"1689178079393"} 2023-07-12 16:07:59,394 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=152, ppid=151, state=RUNNABLE; CloseRegionProcedure 40bb34095ed73242a70fbe098defe2a1, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:07:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=150 2023-07-12 16:07:59,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close 40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:59,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 40bb34095ed73242a70fbe098defe2a1, disabling compactions & flushes 2023-07-12 16:07:59,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:59,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:59,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. after waiting 0 ms 2023-07-12 16:07:59,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:59,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/recovered.edits/3009.seqid, newMaxSeqId=3009, maxSeqId=1 2023-07-12 16:07:59,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1. 2023-07-12 16:07:59,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 40bb34095ed73242a70fbe098defe2a1: 2023-07-12 16:07:59,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed 40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:59,553 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=40bb34095ed73242a70fbe098defe2a1, regionState=CLOSED 2023-07-12 16:07:59,553 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689178079553"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178079553"}]},"ts":"1689178079553"} 2023-07-12 16:07:59,554 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=152, resume processing ppid=151 2023-07-12 16:07:59,554 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=151, state=SUCCESS; CloseRegionProcedure 40bb34095ed73242a70fbe098defe2a1, server=jenkins-hbase5.apache.org,33647,1689177958228 in 159 msec 2023-07-12 16:07:59,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=151, resume processing ppid=150 2023-07-12 16:07:59,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=150, state=SUCCESS; TransitRegionStateProcedure table=testDeleteSnapshots, region=40bb34095ed73242a70fbe098defe2a1, UNASSIGN in 162 msec 2023-07-12 16:07:59,555 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178079555"}]},"ts":"1689178079555"} 2023-07-12 16:07:59,556 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteSnapshots, state=DISABLED in hbase:meta 2023-07-12 16:07:59,557 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testDeleteSnapshots to state=DISABLED 2023-07-12 16:07:59,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, state=SUCCESS; DisableTableProcedure table=testDeleteSnapshots in 170 msec 2023-07-12 16:07:59,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=150 2023-07-12 16:07:59,710 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testDeleteSnapshots completed 2023-07-12 16:07:59,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testDeleteSnapshots 2023-07-12 16:07:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=153, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:07:59,712 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=153, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:07:59,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=153 2023-07-12 16:07:59,712 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=153, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:07:59,716 DEBUG [HFileArchiver-20] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:59,718 DEBUG [HFileArchiver-20] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/recovered.edits] 2023-07-12 16:07:59,721 DEBUG [HFileArchiver-20] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/f1/975a8aa4c4994aa1aec6a8e3d07e9658 2023-07-12 16:07:59,724 DEBUG [HFileArchiver-20] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/recovered.edits/3009.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1/recovered.edits/3009.seqid 2023-07-12 16:07:59,725 DEBUG [HFileArchiver-20] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testDeleteSnapshots/40bb34095ed73242a70fbe098defe2a1 2023-07-12 16:07:59,725 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testDeleteSnapshots regions 2023-07-12 16:07:59,727 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=153, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:07:59,728 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testDeleteSnapshots from hbase:meta 2023-07-12 16:07:59,730 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testDeleteSnapshots' descriptor. 2023-07-12 16:07:59,731 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=153, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:07:59,731 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testDeleteSnapshots' from region states. 2023-07-12 16:07:59,731 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178079731"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:59,735 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:07:59,735 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 40bb34095ed73242a70fbe098defe2a1, NAME => 'testDeleteSnapshots,,1689178065847.40bb34095ed73242a70fbe098defe2a1.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:07:59,735 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testDeleteSnapshots' as deleted. 2023-07-12 16:07:59,736 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteSnapshots","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689178079735"}]},"ts":"9223372036854775807"} 2023-07-12 16:07:59,737 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testDeleteSnapshots state from META 2023-07-12 16:07:59,742 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=153, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteSnapshots 2023-07-12 16:07:59,743 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=153, state=SUCCESS; DeleteTableProcedure table=testDeleteSnapshots in 32 msec 2023-07-12 16:07:59,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=153 2023-07-12 16:07:59,819 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testDeleteSnapshots 2023-07-12 16:07:59,820 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testDeleteSnapshots completed 2023-07-12 16:07:59,832 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testDeleteSnapshots[1] Thread=267 (was 271), OpenFileDescriptor=513 (was 517), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=210 (was 209) - SystemLoadAverage LEAK? -, ProcessCount=161 (was 161), AvailableMemoryMB=6583 (was 6746) 2023-07-12 16:07:59,843 INFO [Listener at localhost/39041] hbase.ResourceChecker(147): before: client.TestAsyncSnapshotAdminApi#testListSnapshots[1] Thread=267, OpenFileDescriptor=513, MaxFileDescriptor=60000, SystemLoadAverage=210, ProcessCount=161, AvailableMemoryMB=6583 2023-07-12 16:07:59,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testListSnapshots', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 16:07:59,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=154, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testListSnapshots 2023-07-12 16:07:59,846 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=154, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 16:07:59,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testListSnapshots" procId is: 154 2023-07-12 16:07:59,847 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=154, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 16:07:59,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=154 2023-07-12 16:07:59,848 DEBUG [HFileArchiver-21] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:07:59,849 DEBUG [HFileArchiver-21] backup.HFileArchiver(153): Directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7 empty. 2023-07-12 16:07:59,849 DEBUG [HFileArchiver-21] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:07:59,849 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testListSnapshots regions 2023-07-12 16:07:59,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=154 2023-07-12 16:08:00,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=154 2023-07-12 16:08:00,258 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/.tabledesc/.tableinfo.0000000001 2023-07-12 16:08:00,259 INFO [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(7675): creating {ENCODED => d83f5c5ab25ffe5289f9e2a40b676bc7, NAME => 'testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testListSnapshots', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f1', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp 2023-07-12 16:08:00,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=154 2023-07-12 16:08:00,674 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(866): Instantiated testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:08:00,674 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1604): Closing d83f5c5ab25ffe5289f9e2a40b676bc7, disabling compactions & flushes 2023-07-12 16:08:00,674 INFO [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1626): Closing region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:00,674 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:00,674 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1714): Acquired close lock on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. after waiting 0 ms 2023-07-12 16:08:00,674 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1724): Updates disabled for region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:00,674 INFO [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1838): Closed testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:00,674 DEBUG [RegionOpenAndInit-testListSnapshots-pool-0] regionserver.HRegion(1558): Region close journal for d83f5c5ab25ffe5289f9e2a40b676bc7: 2023-07-12 16:08:00,676 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=154, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 16:08:00,677 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178080677"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178080677"}]},"ts":"1689178080677"} 2023-07-12 16:08:00,677 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 16:08:00,678 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=154, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 16:08:00,678 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178080678"}]},"ts":"1689178080678"} 2023-07-12 16:08:00,678 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testListSnapshots, state=ENABLING in hbase:meta 2023-07-12 16:08:00,681 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase5.apache.org=0} racks are {/default-rack=0} 2023-07-12 16:08:00,682 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-12 16:08:00,682 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-12 16:08:00,682 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-12 16:08:00,682 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=154, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testListSnapshots, region=d83f5c5ab25ffe5289f9e2a40b676bc7, ASSIGN}] 2023-07-12 16:08:00,683 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=155, ppid=154, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testListSnapshots, region=d83f5c5ab25ffe5289f9e2a40b676bc7, ASSIGN 2023-07-12 16:08:00,684 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=155, ppid=154, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testListSnapshots, region=d83f5c5ab25ffe5289f9e2a40b676bc7, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,33647,1689177958228; forceNewPlan=false, retain=false 2023-07-12 16:08:00,834 INFO [jenkins-hbase5:39879] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-12 16:08:00,835 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=155 updating hbase:meta row=d83f5c5ab25ffe5289f9e2a40b676bc7, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:00,835 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178080835"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178080835"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178080835"}]},"ts":"1689178080835"} 2023-07-12 16:08:00,836 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=155, state=RUNNABLE; OpenRegionProcedure d83f5c5ab25ffe5289f9e2a40b676bc7, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:08:00,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=154 2023-07-12 16:08:00,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:00,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d83f5c5ab25ffe5289f9e2a40b676bc7, NAME => 'testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.', STARTKEY => '', ENDKEY => ''} 2023-07-12 16:08:00,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testListSnapshots d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:00,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 16:08:00,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:00,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:00,992 INFO [StoreOpener-d83f5c5ab25ffe5289f9e2a40b676bc7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:00,993 DEBUG [StoreOpener-d83f5c5ab25ffe5289f9e2a40b676bc7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1 2023-07-12 16:08:00,993 DEBUG [StoreOpener-d83f5c5ab25ffe5289f9e2a40b676bc7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1 2023-07-12 16:08:00,993 INFO [StoreOpener-d83f5c5ab25ffe5289f9e2a40b676bc7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d83f5c5ab25ffe5289f9e2a40b676bc7 columnFamilyName f1 2023-07-12 16:08:00,994 INFO [StoreOpener-d83f5c5ab25ffe5289f9e2a40b676bc7-1] regionserver.HStore(310): Store=d83f5c5ab25ffe5289f9e2a40b676bc7/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 16:08:00,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:00,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:00,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:00,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 16:08:01,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened d83f5c5ab25ffe5289f9e2a40b676bc7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67312404, jitterRate=0.003032982349395752}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 16:08:01,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for d83f5c5ab25ffe5289f9e2a40b676bc7: 2023-07-12 16:08:01,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7., pid=156, masterSystemTime=1689178080987 2023-07-12 16:08:01,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:01,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:01,001 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=155 updating hbase:meta row=d83f5c5ab25ffe5289f9e2a40b676bc7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:01,001 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178081001"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689178081001"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689178081001"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689178081001"}]},"ts":"1689178081001"} 2023-07-12 16:08:01,003 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=155 2023-07-12 16:08:01,003 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=155, state=SUCCESS; OpenRegionProcedure d83f5c5ab25ffe5289f9e2a40b676bc7, server=jenkins-hbase5.apache.org,33647,1689177958228 in 166 msec 2023-07-12 16:08:01,004 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=154 2023-07-12 16:08:01,004 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=154, state=SUCCESS; TransitRegionStateProcedure table=testListSnapshots, region=d83f5c5ab25ffe5289f9e2a40b676bc7, ASSIGN in 321 msec 2023-07-12 16:08:01,004 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=154, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 16:08:01,004 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178081004"}]},"ts":"1689178081004"} 2023-07-12 16:08:01,005 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testListSnapshots, state=ENABLED in hbase:meta 2023-07-12 16:08:01,009 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=154, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testListSnapshots execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 16:08:01,010 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=154, state=SUCCESS; CreateTableProcedure table=testListSnapshots in 1.1640 sec 2023-07-12 16:08:01,487 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testListSnapshots' 2023-07-12 16:08:01,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=154 2023-07-12 16:08:01,952 INFO [Listener at localhost/39041] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testListSnapshots, procId: 154 completed 2023-07-12 16:08:01,952 DEBUG [Listener at localhost/39041] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testListSnapshots get assigned. Timeout = 60000ms 2023-07-12 16:08:01,952 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:08:01,954 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3484): All regions for table testListSnapshots assigned to meta. Checking AM states. 2023-07-12 16:08:01,955 INFO [Listener at localhost/39041] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 16:08:01,955 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(3504): All regions for table testListSnapshots assigned. 2023-07-12 16:08:01,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38637] ipc.CallRunner(144): callId: 27188 service: ClientService methodName: Mutate size: 144 connection: 172.31.10.131:41838 deadline: 1689178141955, exception=org.apache.hadoop.hbase.NotServingRegionException: testListSnapshots,,1689178012279.6b686599b729501fea84a4a7c0fcc5bc. is not online on jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:03,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178083108 (current time:1689178083108). 2023-07-12 16:08:03,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:08:03,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x1202e285 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:08:03,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@348f0c9a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:08:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:08:03,117 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:51460, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:08:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1202e285 to 127.0.0.1:55421 2023-07-12 16:08:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:03,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:08:03,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:03,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=157, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:03,121 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=157, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:03,121 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=157, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178683121 2023-07-12 16:08:03,121 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testListSnapshots 2023-07-12 16:08:03,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:03,123 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=158, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:03,123 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=157, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:03,124 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=157, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE in 3 msec 2023-07-12 16:08:03,124 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=158, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:03,125 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=158, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED; timeout=600000, timestamp=1689178683125 2023-07-12 16:08:03,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:03,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:03,138 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName1 was in running list but was completed. Accepting new attempt. 2023-07-12 16:08:03,138 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName1 2023-07-12 16:08:03,138 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName1' 2023-07-12 16:08:03,138 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:03,138 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName1' starting 'acquire' 2023-07-12 16:08:03,138 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName1', kicking off acquire phase on members. 2023-07-12 16:08:03,138 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,138 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:08:03,139 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:03,139 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:03,140 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:03,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:03,140 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,140 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:08:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:08:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:08:03,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:08:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:08:03,141 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testListSnapshots type FLUSH 2023-07-12 16:08:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:08:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:08:03,141 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName1 from table testListSnapshots type FLUSH 2023-07-12 16:08:03,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:08:03,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:08:03,141 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:08:03,141 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:03,142 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:08:03,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName1 is still present, removing 2023-07-12 16:08:03,142 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:08:03,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName1 2023-07-12 16:08:03,142 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:08:03,142 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName1' with timeout 300000ms 2023-07-12 16:08:03,142 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:03,142 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName1' starting 'acquire' stage 2023-07-12 16:08:03,143 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName1' locally acquired 2023-07-12 16:08:03,143 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName1) in zk 2023-07-12 16:08:03,145 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,145 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,145 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:03,145 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:08:03,145 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:03,146 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,146 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:08:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:08:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:03,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName1' on coordinator 2023-07-12 16:08:03,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28daa4d9[Count = 0] remaining members to acquire global barrier 2023-07-12 16:08:03,147 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName1' starting 'in-barrier' execution. 2023-07-12 16:08:03,147 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,148 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,148 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,149 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:08:03,149 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName1' received 'reached' from coordinator. 2023-07-12 16:08:03,149 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:08:03,149 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,149 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:08:03,149 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:08:03,149 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:08:03,149 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:08:03,149 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:03,149 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. started... 2023-07-12 16:08:03,150 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2745): Flushing d83f5c5ab25ffe5289f9e2a40b676bc7 1/1 column families, dataSize=93.75 KB heapSize=304.94 KB 2023-07-12 16:08:03,151 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:08:03,151 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:03,151 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:08:03,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:03,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:03,560 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=93.75 KB at sequenceid=3004 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/.tmp/f1/3e76e7ea8cf04e8ea7733542e7be3a9f 2023-07-12 16:08:03,566 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/.tmp/f1/3e76e7ea8cf04e8ea7733542e7be3a9f as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f 2023-07-12 16:08:03,570 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f, entries=3000, sequenceid=3004, filesize=106.8 K 2023-07-12 16:08:03,571 INFO [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.75 KB/96000, heapSize ~304.92 KB/312240, currentSize=0 B/0 for d83f5c5ab25ffe5289f9e2a40b676bc7 in 421ms, sequenceid=3004, compaction requested=false 2023-07-12 16:08:03,572 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for d83f5c5ab25ffe5289f9e2a40b676bc7: 2023-07-12 16:08:03,572 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.' region-info for snapshot=snapshotName1 2023-07-12 16:08:03,572 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:08:03,572 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f] hfiles 2023-07-12 16:08:03,572 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f for snapshot=snapshotName1 2023-07-12 16:08:03,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:03,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:03,978 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. completed. 2023-07-12 16:08:03,978 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:03,978 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:08:03,978 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:08:03,978 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,978 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName1' locally completed 2023-07-12 16:08:03,978 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName1' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:08:03,981 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName1' has notified controller of completion 2023-07-12 16:08:03,981 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,981 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:03,981 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName1' completed. 2023-07-12 16:08:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:08:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:08:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName1' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:08:03,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName1', counting down latch. Waiting for 0 more 2023-07-12 16:08:03,985 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName1' execution completed 2023-07-12 16:08:03,985 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:08:03,985 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:08:03,985 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName1 2023-07-12 16:08:03,985 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:08:03,987 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,987 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,987 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,987 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,987 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:03,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:03,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:03,987 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:08:03,988 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:08:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,989 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:03,990 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName1 2023-07-12 16:08:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,992 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:03,992 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,992 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:03,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:03,993 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:03,992 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:03,993 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName1 2023-07-12 16:08:03,992 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:08:03,993 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName1 2023-07-12 16:08:03,993 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:03,993 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:03,993 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName1 2023-07-12 16:08:03,994 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:03,994 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,994 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:03,994 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName1 2023-07-12 16:08:03,994 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName1 2023-07-12 16:08:03,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:03,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:03,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:03,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:03,994 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:08:04,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:04,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:04,412 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName1 2023-07-12 16:08:05,076 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 16:08:05,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:05,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:05,235 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName1 of table testListSnapshots completed 2023-07-12 16:08:05,235 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:08:05,235 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName1 2023-07-12 16:08:05,235 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName1 C_M_SNAPSHOT_TABLE on table testListSnapshots at 1689178083121Consolidate snapshot: snapshotName1 at 1689178083993 (+872 ms)Loading Region manifests for snapshotName1 at 1689178083993Writing data manifest for snapshotName1 at 1689178083997 (+4 ms)Verifying snapshot: snapshotName1 at 1689178084405 (+408 ms)Snapshot snapshotName1 of table testListSnapshots completed at 1689178085234 (+829 ms) 2023-07-12 16:08:05,236 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=158, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:05,238 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=158, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED in 2.1140 sec 2023-07-12 16:08:07,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:07,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName1 table=testListSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:08:07,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:07,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178087191 (current time:1689178087191). 2023-07-12 16:08:07,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:08:07,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x7154da63 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:08:07,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74c71806, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:08:07,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:08:07,199 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:08:07,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7154da63 to 127.0.0.1:55421 2023-07-12 16:08:07,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:07,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:08:07,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:07,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=159, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:07,203 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=159, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:07,204 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=159, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178687204 2023-07-12 16:08:07,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:07,204 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testListSnapshots 2023-07-12 16:08:07,205 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=159, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:07,205 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=160, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:07,205 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=159, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE in 3 msec 2023-07-12 16:08:07,206 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=160, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:07,206 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=160, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED; timeout=600000, timestamp=1689178687206 2023-07-12 16:08:07,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:07,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:07,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:07,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:07,615 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName2 was in running list but was completed. Accepting new attempt. 2023-07-12 16:08:07,615 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName2 2023-07-12 16:08:07,615 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName2' 2023-07-12 16:08:07,615 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:07,615 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName2' starting 'acquire' 2023-07-12 16:08:07,615 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName2', kicking off acquire phase on members. 2023-07-12 16:08:07,616 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:07,616 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:08:07,617 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:07,617 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:07,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:07,617 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:07,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:07,618 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:07,618 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:08:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:08:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:08:07,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:07,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:08:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:08:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:08:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:08:07,619 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testListSnapshots type FLUSH 2023-07-12 16:08:07,619 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName2 from table testListSnapshots type FLUSH 2023-07-12 16:08:07,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:08:07,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:08:07,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName2 is still present, removing 2023-07-12 16:08:07,619 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:08:07,619 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:07,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName2 2023-07-12 16:08:07,620 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:08:07,620 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName2' with timeout 300000ms 2023-07-12 16:08:07,620 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:07,620 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:08:07,620 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:08:07,620 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName2' starting 'acquire' stage 2023-07-12 16:08:07,620 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName2' locally acquired 2023-07-12 16:08:07,620 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName2) in zk 2023-07-12 16:08:07,622 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,622 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,622 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:07,622 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:07,623 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:08:07,623 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,623 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:08:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:08:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:07,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName2' on coordinator 2023-07-12 16:08:07,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa58c2[Count = 0] remaining members to acquire global barrier 2023-07-12 16:08:07,624 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName2' starting 'in-barrier' execution. 2023-07-12 16:08:07,624 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,626 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,626 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:07,627 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:07,627 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:08:07,627 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:08:07,627 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName2' received 'reached' from coordinator. 2023-07-12 16:08:07,627 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:08:07,627 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:08:07,627 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:08:07,627 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:08:07,627 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:07,627 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. started... 2023-07-12 16:08:07,627 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for d83f5c5ab25ffe5289f9e2a40b676bc7: 2023-07-12 16:08:07,627 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.' region-info for snapshot=snapshotName2 2023-07-12 16:08:07,627 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:08:07,627 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f] hfiles 2023-07-12 16:08:07,627 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f for snapshot=snapshotName2 2023-07-12 16:08:07,628 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:08:07,628 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:07,628 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:08:07,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:07,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:08,036 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. completed. 2023-07-12 16:08:08,036 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:08,037 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:08:08,037 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:08:08,037 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,037 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName2' locally completed 2023-07-12 16:08:08,037 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName2' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:08:08,038 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,038 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName2' has notified controller of completion 2023-07-12 16:08:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:08,038 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:08,038 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName2' completed. 2023-07-12 16:08:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:08:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:08:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName2' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:08:08,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName2', counting down latch. Waiting for 0 more 2023-07-12 16:08:08,042 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName2' execution completed 2023-07-12 16:08:08,042 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:08:08,042 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:08:08,042 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName2 2023-07-12 16:08:08,042 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:08:08,044 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,044 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:08,044 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,044 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,044 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:08,044 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:08:08,045 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:08:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,046 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:08,046 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName2 2023-07-12 16:08:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,053 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,053 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:08,053 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:08:08,053 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:08,053 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:08,053 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:08,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:08,054 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName2 2023-07-12 16:08:08,053 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName2 2023-07-12 16:08:08,053 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:08,054 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName2 2023-07-12 16:08:08,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:08,053 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:08,054 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:08,054 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:08,055 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:08,055 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName2 2023-07-12 16:08:08,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:08,055 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName2 2023-07-12 16:08:08,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:08,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:08,055 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:08:08,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:08,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:08,470 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName2 2023-07-12 16:08:09,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:09,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:09,287 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName2 of table testListSnapshots completed 2023-07-12 16:08:09,288 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:08:09,288 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName2 2023-07-12 16:08:09,288 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName2 C_M_SNAPSHOT_TABLE on table testListSnapshots at 1689178087204Consolidate snapshot: snapshotName2 at 1689178088054 (+850 ms)Loading Region manifests for snapshotName2 at 1689178088054Writing data manifest for snapshotName2 at 1689178088058 (+4 ms)Verifying snapshot: snapshotName2 at 1689178088463 (+405 ms)Snapshot snapshotName2 of table testListSnapshots completed at 1689178089287 (+824 ms) 2023-07-12 16:08:09,289 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=160, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:09,290 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=160, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED in 2.0850 sec 2023-07-12 16:08:11,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:11,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName2 table=testListSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:08:11,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:11,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689178091271 (current time:1689178091271). 2023-07-12 16:08:11,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 16:08:11,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(139): Connect 0x67070b76 to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:08:11,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30d74237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:08:11,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 16:08:11,280 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55020, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 16:08:11,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67070b76 to 127.0.0.1:55421 2023-07-12 16:08:11,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:11,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 16:08:11,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:11,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:11,284 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=161, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:11,284 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=161, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE; timeout=600000, timestamp=1689178691284 2023-07-12 16:08:11,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(653): Started snapshot: { ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } 2023-07-12 16:08:11,284 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot snapshotName3 C_M_SNAPSHOT_TABLE on table testListSnapshots 2023-07-12 16:08:11,287 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=161, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE 2023-07-12 16:08:11,287 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=162, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:11,287 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=EXCLUSIVE in 4 msec 2023-07-12 16:08:11,288 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=162, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:11,288 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=162, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED; timeout=600000, timestamp=1689178691288 2023-07-12 16:08:11,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:11,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:11,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:11,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:11,697 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(143): Procedure snapshotName3 was in running list but was completed. Accepting new attempt. 2023-07-12 16:08:11,697 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure snapshotName3 2023-07-12 16:08:11,697 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'snapshotName3' 2023-07-12 16:08:11,697 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:11,698 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'snapshotName3' starting 'acquire' 2023-07-12 16:08:11,698 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'snapshotName3', kicking off acquire phase on members. 2023-07-12 16:08:11,698 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:11,698 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:08:11,699 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:11,699 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:11,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:11,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:11,699 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:08:11,700 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:11,700 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 16:08:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:08:11,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:11,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:08:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:08:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 51 2023-07-12 16:08:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:08:11,701 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName3 from table testListSnapshots type FLUSH 2023-07-12 16:08:11,701 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot snapshotName3 from table testListSnapshots type FLUSH 2023-07-12 16:08:11,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName3 is still present, removing 2023-07-12 16:08:11,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName3 2023-07-12 16:08:11,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc snapshotName3 is still present, removing 2023-07-12 16:08:11,701 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName3' with timeout 300000ms 2023-07-12 16:08:11,701 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:11,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:snapshotName3 2023-07-12 16:08:11,702 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName3' starting 'acquire' stage 2023-07-12 16:08:11,702 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'snapshotName3' with timeout 300000ms 2023-07-12 16:08:11,702 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 16:08:11,702 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName3' locally acquired 2023-07-12 16:08:11,702 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure (snapshotName3) in zk 2023-07-12 16:08:11,702 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'snapshotName3' starting 'acquire' stage 2023-07-12 16:08:11,703 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'snapshotName3' locally acquired 2023-07-12 16:08:11,703 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,38637,1689177958270' joining acquired barrier for procedure (snapshotName3) in zk 2023-07-12 16:08:11,705 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,705 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,705 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:11,706 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,706 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:08:11,706 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,706 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'snapshotName3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 16:08:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:08:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:11,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,33647,1689177958228' joining acquired barrier for procedure 'snapshotName3' on coordinator 2023-07-12 16:08:11,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22df4e56[Count = 0] remaining members to acquire global barrier 2023-07-12 16:08:11,707 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'snapshotName3' starting 'in-barrier' execution. 2023-07-12 16:08:11,707 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,709 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,709 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:11,709 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName3' received 'reached' from coordinator. 2023-07-12 16:08:11,709 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'snapshotName3' received 'reached' from coordinator. 2023-07-12 16:08:11,709 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName3' locally completed 2023-07-12 16:08:11,709 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:11,709 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 16:08:11,709 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName3' completed for member 'jenkins-hbase5.apache.org,38637,1689177958270' in zk 2023-07-12 16:08:11,709 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 16:08:11,709 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 16:08:11,709 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:11,709 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. started... 2023-07-12 16:08:11,709 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for d83f5c5ab25ffe5289f9e2a40b676bc7: 2023-07-12 16:08:11,710 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.' region-info for snapshot=snapshotName3 2023-07-12 16:08:11,710 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 16:08:11,710 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f] hfiles 2023-07-12 16:08:11,710 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f for snapshot=snapshotName3 2023-07-12 16:08:11,711 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName3' has notified controller of completion 2023-07-12 16:08:11,711 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:11,711 DEBUG [member: 'jenkins-hbase5.apache.org,38637,1689177958270' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName3' completed. 2023-07-12 16:08:11,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:11,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:12,116 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. completed. 2023-07-12 16:08:12,116 DEBUG [rs(jenkins-hbase5.apache.org,33647,1689177958228)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:12,116 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 16:08:12,116 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 16:08:12,116 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,116 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'snapshotName3' locally completed 2023-07-12 16:08:12,116 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'snapshotName3' completed for member 'jenkins-hbase5.apache.org,33647,1689177958228' in zk 2023-07-12 16:08:12,118 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:12,119 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'snapshotName3' has notified controller of completion 2023-07-12 16:08:12,119 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:12,119 DEBUG [member: 'jenkins-hbase5.apache.org,33647,1689177958228' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'snapshotName3' completed. 2023-07-12 16:08:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:08:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:08:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'snapshotName3' member 'jenkins-hbase5.apache.org,33647,1689177958228': 2023-07-12 16:08:12,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,33647,1689177958228' released barrier for procedure'snapshotName3', counting down latch. Waiting for 0 more 2023-07-12 16:08:12,124 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'snapshotName3' execution completed 2023-07-12 16:08:12,124 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 16:08:12,124 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 16:08:12,124 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:snapshotName3 2023-07-12 16:08:12,124 INFO [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure snapshotName3including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 16:08:12,132 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,132 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,132 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:12,132 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,132 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 16:08:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 16:08:12,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:12,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 16:08:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,134 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:08:12,135 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 16:08:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:08:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,137 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 16:08:12,137 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----snapshotName3 2023-07-12 16:08:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,141 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:12,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:12,141 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 16:08:12,141 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:12,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 16:08:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/snapshotName3 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/snapshotName3 2023-07-12 16:08:12,142 DEBUG [(jenkins-hbase5.apache.org,39879,1689177956989)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 16:08:12,142 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/snapshotName3 2023-07-12 16:08:12,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:12,143 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 16:08:12,142 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for snapshotName3 2023-07-12 16:08:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:12,143 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snapshotName3 2023-07-12 16:08:12,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 16:08:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 16:08:12,145 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:08:12,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:12,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:12,567 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/snapshotName3 2023-07-12 16:08:13,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:13,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(404): Snapshoting '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' is still in progress! 2023-07-12 16:08:13,384 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snapshotName3 of table testListSnapshots completed 2023-07-12 16:08:13,384 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:08:13,384 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.hbase-snapshot/.tmp/snapshotName3 2023-07-12 16:08:13,384 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot snapshotName3 C_M_SNAPSHOT_TABLE on table testListSnapshots at 1689178091284Consolidate snapshot: snapshotName3 at 1689178092143 (+859 ms)Loading Region manifests for snapshotName3 at 1689178092143Writing data manifest for snapshotName3 at 1689178092148 (+5 ms)Verifying snapshot: snapshotName3 at 1689178092557 (+409 ms)Snapshot snapshotName3 of table testListSnapshots completed at 1689178093384 (+827 ms) 2023-07-12 16:08:13,385 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=162, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED 2023-07-12 16:08:13,386 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=162, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testListSnapshots, type=SHARED in 2.0990 sec 2023-07-12 16:08:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 } is done 2023-07-12 16:08:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(401): Snapshot '{ ss=snapshotName3 table=testListSnapshots type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 16:08:15,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName1" table: "testListSnapshots" creation_time: 1689178083108 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:08:15,394 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName3" table: "testListSnapshots" creation_time: 1689178091271 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:08:15,394 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(751): Client=jenkins//172.31.10.131 delete name: "snapshotName2" table: "testListSnapshots" creation_time: 1689178087191 type: FLUSH version: 2 owner: "" ttl: 0 max_file_size: 0 2023-07-12 16:08:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName1 2023-07-12 16:08:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName3 2023-07-12 16:08:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] snapshot.SnapshotManager(333): Deleting snapshot: snapshotName2 2023-07-12 16:08:15,399 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.HMaster$11(2418): Client=jenkins//172.31.10.131 disable testListSnapshots 2023-07-12 16:08:15,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=163, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testListSnapshots 2023-07-12 16:08:15,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=163 2023-07-12 16:08:15,401 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178095401"}]},"ts":"1689178095401"} 2023-07-12 16:08:15,402 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testListSnapshots, state=DISABLING in hbase:meta 2023-07-12 16:08:15,403 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testListSnapshots to state=DISABLING 2023-07-12 16:08:15,404 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=164, ppid=163, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testListSnapshots, region=d83f5c5ab25ffe5289f9e2a40b676bc7, UNASSIGN}] 2023-07-12 16:08:15,405 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=163, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testListSnapshots, region=d83f5c5ab25ffe5289f9e2a40b676bc7, UNASSIGN 2023-07-12 16:08:15,405 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=d83f5c5ab25ffe5289f9e2a40b676bc7, regionState=CLOSING, regionLocation=jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:15,405 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178095405"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689178095405"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689178095405"}]},"ts":"1689178095405"} 2023-07-12 16:08:15,406 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=165, ppid=164, state=RUNNABLE; CloseRegionProcedure d83f5c5ab25ffe5289f9e2a40b676bc7, server=jenkins-hbase5.apache.org,33647,1689177958228}] 2023-07-12 16:08:15,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=163 2023-07-12 16:08:15,557 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(111): Close d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:15,557 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing d83f5c5ab25ffe5289f9e2a40b676bc7, disabling compactions & flushes 2023-07-12 16:08:15,557 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:15,557 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:15,557 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. after waiting 0 ms 2023-07-12 16:08:15,557 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:15,564 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/recovered.edits/3009.seqid, newMaxSeqId=3009, maxSeqId=1 2023-07-12 16:08:15,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7. 2023-07-12 16:08:15,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for d83f5c5ab25ffe5289f9e2a40b676bc7: 2023-07-12 16:08:15,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.UnassignRegionHandler(149): Closed d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:15,567 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=d83f5c5ab25ffe5289f9e2a40b676bc7, regionState=CLOSED 2023-07-12 16:08:15,567 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.","families":{"info":[{"qualifier":"regioninfo","vlen":51,"tag":[],"timestamp":"1689178095567"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689178095567"}]},"ts":"1689178095567"} 2023-07-12 16:08:15,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=165, resume processing ppid=164 2023-07-12 16:08:15,569 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=164, state=SUCCESS; CloseRegionProcedure d83f5c5ab25ffe5289f9e2a40b676bc7, server=jenkins-hbase5.apache.org,33647,1689177958228 in 162 msec 2023-07-12 16:08:15,570 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=164, resume processing ppid=163 2023-07-12 16:08:15,570 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=163, state=SUCCESS; TransitRegionStateProcedure table=testListSnapshots, region=d83f5c5ab25ffe5289f9e2a40b676bc7, UNASSIGN in 165 msec 2023-07-12 16:08:15,571 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689178095571"}]},"ts":"1689178095571"} 2023-07-12 16:08:15,571 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testListSnapshots, state=DISABLED in hbase:meta 2023-07-12 16:08:15,573 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testListSnapshots to state=DISABLED 2023-07-12 16:08:15,575 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=163, state=SUCCESS; DisableTableProcedure table=testListSnapshots in 175 msec 2023-07-12 16:08:15,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=163 2023-07-12 16:08:15,720 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DISABLE, Table Name: default:testListSnapshots completed 2023-07-12 16:08:15,721 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.HMaster$5(2228): Client=jenkins//172.31.10.131 delete testListSnapshots 2023-07-12 16:08:15,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] procedure2.ProcedureExecutor(1029): Stored pid=166, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:08:15,722 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=166, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:08:15,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=166 2023-07-12 16:08:15,723 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=166, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:08:15,726 DEBUG [HFileArchiver-22] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:15,727 DEBUG [HFileArchiver-22] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1, FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/recovered.edits] 2023-07-12 16:08:15,731 DEBUG [HFileArchiver-22] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/f1/3e76e7ea8cf04e8ea7733542e7be3a9f 2023-07-12 16:08:15,734 DEBUG [HFileArchiver-22] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/recovered.edits/3009.seqid to hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/archive/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7/recovered.edits/3009.seqid 2023-07-12 16:08:15,734 DEBUG [HFileArchiver-22] backup.HFileArchiver(596): Deleted hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/.tmp/data/default/testListSnapshots/d83f5c5ab25ffe5289f9e2a40b676bc7 2023-07-12 16:08:15,735 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testListSnapshots regions 2023-07-12 16:08:15,736 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=166, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:08:15,738 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testListSnapshots from hbase:meta 2023-07-12 16:08:15,739 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testListSnapshots' descriptor. 2023-07-12 16:08:15,740 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=166, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:08:15,740 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testListSnapshots' from region states. 2023-07-12 16:08:15,740 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689178095740"}]},"ts":"9223372036854775807"} 2023-07-12 16:08:15,741 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-07-12 16:08:15,741 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d83f5c5ab25ffe5289f9e2a40b676bc7, NAME => 'testListSnapshots,,1689178079845.d83f5c5ab25ffe5289f9e2a40b676bc7.', STARTKEY => '', ENDKEY => ''}] 2023-07-12 16:08:15,742 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testListSnapshots' as deleted. 2023-07-12 16:08:15,742 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testListSnapshots","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689178095742"}]},"ts":"9223372036854775807"} 2023-07-12 16:08:15,743 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testListSnapshots state from META 2023-07-12 16:08:15,747 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=166, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testListSnapshots 2023-07-12 16:08:15,748 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=166, state=SUCCESS; DeleteTableProcedure table=testListSnapshots in 26 msec 2023-07-12 16:08:15,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39879] master.MasterRpcServices(1230): Checking to see if procedure is done pid=166 2023-07-12 16:08:15,830 DEBUG [RPCClient-NioEventLoopGroup-6-4] client.AsyncRegionLocator(157): Clear meta cache for testListSnapshots 2023-07-12 16:08:15,830 INFO [RPCClient-NioEventLoopGroup-6-4] client.RawAsyncHBaseAdmin$TableProcedureBiConsumer(2487): Operation: DELETE, Table Name: default:testListSnapshots completed 2023-07-12 16:08:15,845 INFO [Listener at localhost/39041] hbase.ResourceChecker(175): after: client.TestAsyncSnapshotAdminApi#testListSnapshots[1] Thread=269 (was 267) - Thread LEAK? -, OpenFileDescriptor=518 (was 513) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=213 (was 210) - SystemLoadAverage LEAK? -, ProcessCount=161 (was 161), AvailableMemoryMB=6910 (was 6583) - AvailableMemoryMB LEAK? - 2023-07-12 16:08:15,845 INFO [Listener at localhost/39041] client.AsyncConnectionImpl(207): Connection has been closed by Listener at localhost/39041. 2023-07-12 16:08:15,847 DEBUG [Listener at localhost/39041] client.AsyncConnectionImpl(232): Call stack: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:209) at org.apache.hbase.thirdparty.com.google.common.io.Closeables.close(Closeables.java:79) at org.apache.hadoop.hbase.client.TestAsyncAdminBase.tearDownAfterClass(TestAsyncAdminBase.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 16:08:15,856 DEBUG [Listener at localhost/39041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:15,858 DEBUG [Listener at localhost/39041] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b24fef6 to 127.0.0.1:55421 2023-07-12 16:08:15,858 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 16:08:15,858 INFO [Listener at localhost/39041] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 16:08:15,858 DEBUG [Listener at localhost/39041] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b9833e1 to 127.0.0.1:55421 2023-07-12 16:08:15,859 DEBUG [Listener at localhost/39041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:15,861 DEBUG [Listener at localhost/39041] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 16:08:15,861 DEBUG [Listener at localhost/39041] util.JVMClusterUtil(257): Found active master hash=1784461350, stopped=false 2023-07-12 16:08:15,862 INFO [Listener at localhost/39041] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,34467,1689177958173' ***** 2023-07-12 16:08:15,862 INFO [Listener at localhost/39041] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/39041 2023-07-12 16:08:15,862 DEBUG [M:1;jenkins-hbase5:34467] zookeeper.ReadOnlyZKClient(139): Connect 0x100d24ac to 127.0.0.1:55421 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 16:08:15,863 INFO [Listener at localhost/39041] util.JVMClusterUtil(250): Stopped backup Master 1362112151 is stopped: true 2023-07-12 16:08:15,863 INFO [Listener at localhost/39041] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:08:15,864 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 16:08:15,864 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 16:08:15,864 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:08:15,864 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 16:08:15,864 INFO [Listener at localhost/39041] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 16:08:15,864 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 16:08:15,865 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:08:15,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 16:08:15,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 16:08:15,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 16:08:15,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 16:08:15,887 DEBUG [Listener at localhost/39041] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5892ba3f to 127.0.0.1:55421 2023-07-12 16:08:15,887 DEBUG [Listener at localhost/39041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:15,888 INFO [Listener at localhost/39041] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,33647,1689177958228' ***** 2023-07-12 16:08:15,888 INFO [Listener at localhost/39041] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 16:08:15,888 INFO [Listener at localhost/39041] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,38637,1689177958270' ***** 2023-07-12 16:08:15,888 INFO [Listener at localhost/39041] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 16:08:15,888 INFO [RS:0;jenkins-hbase5:33647] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 16:08:15,889 INFO [RS:0;jenkins-hbase5:33647] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 16:08:15,889 INFO [RS:0;jenkins-hbase5:33647] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 16:08:15,889 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(3305): Received CLOSE for 6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:08:15,889 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:15,889 DEBUG [RS:0;jenkins-hbase5:33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12dde343 to 127.0.0.1:55421 2023-07-12 16:08:15,890 DEBUG [RS:0;jenkins-hbase5:33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:15,890 INFO [RS:1;jenkins-hbase5:38637] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 16:08:15,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 6527c424ba7b867664c691dcf6c1ec65, disabling compactions & flushes 2023-07-12 16:08:15,891 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:08:15,891 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 16:08:15,891 INFO [RS:1;jenkins-hbase5:38637] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 16:08:15,891 INFO [RS:1;jenkins-hbase5:38637] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 16:08:15,891 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:15,891 DEBUG [RS:1;jenkins-hbase5:38637] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21a06c2b to 127.0.0.1:55421 2023-07-12 16:08:15,891 DEBUG [RS:1;jenkins-hbase5:38637] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:15,891 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 16:08:15,893 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1478): Online Regions={6527c424ba7b867664c691dcf6c1ec65=hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65.} 2023-07-12 16:08:15,892 INFO [RS:1;jenkins-hbase5:38637] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 16:08:15,893 INFO [RS:1;jenkins-hbase5:38637] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 16:08:15,893 INFO [RS:1;jenkins-hbase5:38637] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 16:08:15,893 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 16:08:15,891 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 16:08:15,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:08:15,893 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. after waiting 0 ms 2023-07-12 16:08:15,893 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:08:15,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 6527c424ba7b867664c691dcf6c1ec65 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 16:08:15,895 DEBUG [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1504): Waiting on 6527c424ba7b867664c691dcf6c1ec65 2023-07-12 16:08:15,900 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 16:08:15,900 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-07-12 16:08:15,900 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 16:08:15,900 DEBUG [M:1;jenkins-hbase5:34467] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71c30aa7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 16:08:15,900 DEBUG [M:1;jenkins-hbase5:34467] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21d7d69e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 16:08:15,900 INFO [M:1;jenkins-hbase5:34467] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,34467,1689177958173 2023-07-12 16:08:15,900 DEBUG [M:1;jenkins-hbase5:34467] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x100d24ac to 127.0.0.1:55421 2023-07-12 16:08:15,901 DEBUG [M:1;jenkins-hbase5:34467] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:15,901 INFO [M:1;jenkins-hbase5:34467] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,34467,1689177958173; all regions closed. 2023-07-12 16:08:15,901 DEBUG [M:1;jenkins-hbase5:34467] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:15,901 INFO [M:1;jenkins-hbase5:34467] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 16:08:15,901 DEBUG [M:1;jenkins-hbase5:34467] master.HMaster(1512): Stopping service threads 2023-07-12 16:08:15,911 INFO [M:1;jenkins-hbase5:34467] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:34467 2023-07-12 16:08:15,912 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 16:08:15,912 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 16:08:15,912 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 16:08:15,912 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 16:08:15,912 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 16:08:15,912 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=39.46 KB heapSize=68.41 KB 2023-07-12 16:08:15,924 DEBUG [M:1;jenkins-hbase5:34467] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,34467,1689177958173 already deleted, retry=false 2023-07-12 16:08:15,926 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 16:08:15,926 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,34467,1689177958173 2023-07-12 16:08:15,926 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 16:08:15,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65/.tmp/info/262e44601d7948aca61601dbf3f012d9 2023-07-12 16:08:15,957 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 16:08:15,957 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 16:08:15,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65/.tmp/info/262e44601d7948aca61601dbf3f012d9 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65/info/262e44601d7948aca61601dbf3f012d9 2023-07-12 16:08:15,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65/info/262e44601d7948aca61601dbf3f012d9, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 16:08:15,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 6527c424ba7b867664c691dcf6c1ec65 in 78ms, sequenceid=6, compaction requested=false 2023-07-12 16:08:15,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/namespace/6527c424ba7b867664c691dcf6c1ec65/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 16:08:15,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:08:15,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 6527c424ba7b867664c691dcf6c1ec65: 2023-07-12 16:08:15,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689177961673.6527c424ba7b867664c691dcf6c1ec65. 2023-07-12 16:08:16,026 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 16:08:16,026 INFO [M:1;jenkins-hbase5:34467] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,34467,1689177958173; zookeeper connection closed. 2023-07-12 16:08:16,026 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:34467-0x101c5e4e98d0001, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 16:08:16,095 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,33647,1689177958228; all regions closed. 2023-07-12 16:08:16,100 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 16:08:16,102 DEBUG [RS:0;jenkins-hbase5:33647] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/oldWALs 2023-07-12 16:08:16,102 INFO [RS:0;jenkins-hbase5:33647] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C33647%2C1689177958228:(num 1689177960797) 2023-07-12 16:08:16,102 DEBUG [RS:0;jenkins-hbase5:33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:16,102 INFO [RS:0;jenkins-hbase5:33647] regionserver.LeaseManager(133): Closed leases 2023-07-12 16:08:16,102 INFO [RS:0;jenkins-hbase5:33647] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 16:08:16,103 INFO [RS:0;jenkins-hbase5:33647] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 16:08:16,103 INFO [RS:0;jenkins-hbase5:33647] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 16:08:16,103 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 16:08:16,103 INFO [RS:0;jenkins-hbase5:33647] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 16:08:16,104 INFO [RS:0;jenkins-hbase5:33647] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:33647 2023-07-12 16:08:16,105 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:16,105 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 16:08:16,105 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,33647,1689177958228 2023-07-12 16:08:16,105 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 16:08:16,106 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 16:08:16,107 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,33647,1689177958228] 2023-07-12 16:08:16,108 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,33647,1689177958228; numProcessing=1 2023-07-12 16:08:16,110 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,33647,1689177958228 already deleted, retry=false 2023-07-12 16:08:16,110 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,33647,1689177958228 expired; onlineServers=1 2023-07-12 16:08:16,208 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 16:08:16,208 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:33647-0x101c5e4e98d0002, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 16:08:16,208 INFO [RS:0;jenkins-hbase5:33647] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,33647,1689177958228; zookeeper connection closed. 2023-07-12 16:08:16,208 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@33d7eeaf] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@33d7eeaf 2023-07-12 16:08:16,301 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 16:08:16,368 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.46 KB at sequenceid=193 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/.tmp/info/e3d09ea7fa83478a80ddd35344ef8723 2023-07-12 16:08:16,373 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e3d09ea7fa83478a80ddd35344ef8723 2023-07-12 16:08:16,501 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 16:08:16,645 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 16:08:16,645 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 16:08:16,701 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 16:08:16,781 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.14 KB at sequenceid=193 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/.tmp/rep_barrier/b6a7a32d204046adb5c62afa5094dd58 2023-07-12 16:08:16,786 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b6a7a32d204046adb5c62afa5094dd58 2023-07-12 16:08:16,902 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 16:08:17,102 DEBUG [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 16:08:17,193 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.85 KB at sequenceid=193 (bloomFilter=false), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/.tmp/table/c5411716ff7a472292ca2e8c7d8c272b 2023-07-12 16:08:17,198 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c5411716ff7a472292ca2e8c7d8c272b 2023-07-12 16:08:17,199 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/.tmp/info/e3d09ea7fa83478a80ddd35344ef8723 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/info/e3d09ea7fa83478a80ddd35344ef8723 2023-07-12 16:08:17,203 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e3d09ea7fa83478a80ddd35344ef8723 2023-07-12 16:08:17,204 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/info/e3d09ea7fa83478a80ddd35344ef8723, entries=38, sequenceid=193, filesize=8.6 K 2023-07-12 16:08:17,204 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/.tmp/rep_barrier/b6a7a32d204046adb5c62afa5094dd58 as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/rep_barrier/b6a7a32d204046adb5c62afa5094dd58 2023-07-12 16:08:17,209 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b6a7a32d204046adb5c62afa5094dd58 2023-07-12 16:08:17,209 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/rep_barrier/b6a7a32d204046adb5c62afa5094dd58, entries=12, sequenceid=193, filesize=6.0 K 2023-07-12 16:08:17,210 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/.tmp/table/c5411716ff7a472292ca2e8c7d8c272b as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/table/c5411716ff7a472292ca2e8c7d8c272b 2023-07-12 16:08:17,214 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c5411716ff7a472292ca2e8c7d8c272b 2023-07-12 16:08:17,214 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/table/c5411716ff7a472292ca2e8c7d8c272b, entries=20, sequenceid=193, filesize=6.3 K 2023-07-12 16:08:17,214 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~39.46 KB/40406, heapSize ~68.36 KB/70000, currentSize=0 B/0 for 1588230740 in 1302ms, sequenceid=193, compaction requested=false 2023-07-12 16:08:17,222 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/data/hbase/meta/1588230740/recovered.edits/196.seqid, newMaxSeqId=196, maxSeqId=1 2023-07-12 16:08:17,222 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 16:08:17,223 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 16:08:17,223 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 16:08:17,223 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 16:08:17,302 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,38637,1689177958270; all regions closed. 2023-07-12 16:08:17,305 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/WALs/jenkins-hbase5.apache.org,38637,1689177958270/jenkins-hbase5.apache.org%2C38637%2C1689177958270.meta.1689177961403.meta not finished, retry = 0 2023-07-12 16:08:17,408 DEBUG [RS:1;jenkins-hbase5:38637] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/oldWALs 2023-07-12 16:08:17,409 INFO [RS:1;jenkins-hbase5:38637] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C38637%2C1689177958270.meta:.meta(num 1689177961403) 2023-07-12 16:08:17,414 DEBUG [RS:1;jenkins-hbase5:38637] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/oldWALs 2023-07-12 16:08:17,414 INFO [RS:1;jenkins-hbase5:38637] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C38637%2C1689177958270:(num 1689177960797) 2023-07-12 16:08:17,415 DEBUG [RS:1;jenkins-hbase5:38637] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:17,415 INFO [RS:1;jenkins-hbase5:38637] regionserver.LeaseManager(133): Closed leases 2023-07-12 16:08:17,415 INFO [RS:1;jenkins-hbase5:38637] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 16:08:17,415 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 16:08:17,416 INFO [RS:1;jenkins-hbase5:38637] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:38637 2023-07-12 16:08:17,418 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 16:08:17,418 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,38637,1689177958270 2023-07-12 16:08:17,419 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,38637,1689177958270] 2023-07-12 16:08:17,419 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,38637,1689177958270; numProcessing=2 2023-07-12 16:08:17,420 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,38637,1689177958270 already deleted, retry=false 2023-07-12 16:08:17,420 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,38637,1689177958270 expired; onlineServers=0 2023-07-12 16:08:17,420 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,39879,1689177956989' ***** 2023-07-12 16:08:17,420 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 16:08:17,421 DEBUG [M:0;jenkins-hbase5:39879] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b61b21d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 16:08:17,421 INFO [M:0;jenkins-hbase5:39879] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,39879,1689177956989 2023-07-12 16:08:17,421 INFO [M:0;jenkins-hbase5:39879] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,39879,1689177956989; all regions closed. 2023-07-12 16:08:17,421 DEBUG [M:0;jenkins-hbase5:39879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 16:08:17,421 DEBUG [M:0;jenkins-hbase5:39879] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 16:08:17,421 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 16:08:17,421 DEBUG [M:0;jenkins-hbase5:39879] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 16:08:17,421 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689177960523] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689177960523,5,FailOnTimeoutGroup] 2023-07-12 16:08:17,421 INFO [M:0;jenkins-hbase5:39879] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 16:08:17,421 INFO [M:0;jenkins-hbase5:39879] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 16:08:17,421 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689177960522] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689177960522,5,FailOnTimeoutGroup] 2023-07-12 16:08:17,421 INFO [M:0;jenkins-hbase5:39879] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [] on shutdown 2023-07-12 16:08:17,422 DEBUG [M:0;jenkins-hbase5:39879] master.HMaster(1512): Stopping service threads 2023-07-12 16:08:17,422 INFO [M:0;jenkins-hbase5:39879] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 16:08:17,422 ERROR [M:0;jenkins-hbase5:39879] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-20,5,PEWorkerGroup] Thread[HFileArchiver-21,5,PEWorkerGroup] Thread[HFileArchiver-22,5,PEWorkerGroup] 2023-07-12 16:08:17,423 INFO [M:0;jenkins-hbase5:39879] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 16:08:17,423 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 16:08:17,423 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 16:08:17,423 DEBUG [M:0;jenkins-hbase5:39879] zookeeper.ZKUtil(398): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 16:08:17,423 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 16:08:17,423 WARN [M:0;jenkins-hbase5:39879] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 16:08:17,424 INFO [M:0;jenkins-hbase5:39879] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 16:08:17,424 INFO [M:0;jenkins-hbase5:39879] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 16:08:17,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 16:08:17,424 DEBUG [M:0;jenkins-hbase5:39879] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 16:08:17,424 INFO [M:0;jenkins-hbase5:39879] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 16:08:17,424 DEBUG [M:0;jenkins-hbase5:39879] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 16:08:17,424 DEBUG [M:0;jenkins-hbase5:39879] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 16:08:17,424 DEBUG [M:0;jenkins-hbase5:39879] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 16:08:17,425 INFO [M:0;jenkins-hbase5:39879] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=486.68 KB heapSize=589.61 KB 2023-07-12 16:08:17,519 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 16:08:17,520 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): regionserver:38637-0x101c5e4e98d0003, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 16:08:17,519 INFO [RS:1;jenkins-hbase5:38637] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,38637,1689177958270; zookeeper connection closed. 2023-07-12 16:08:17,520 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@700ffa91] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@700ffa91 2023-07-12 16:08:17,520 INFO [Listener at localhost/39041] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 2 regionserver(s) complete 2023-07-12 16:08:17,854 INFO [M:0;jenkins-hbase5:39879] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=486.68 KB at sequenceid=1262 (bloomFilter=true), to=hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9f7d80ed87754f249fcfa813ef8b329f 2023-07-12 16:08:17,858 INFO [M:0;jenkins-hbase5:39879] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9f7d80ed87754f249fcfa813ef8b329f 2023-07-12 16:08:17,859 DEBUG [M:0;jenkins-hbase5:39879] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9f7d80ed87754f249fcfa813ef8b329f as hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9f7d80ed87754f249fcfa813ef8b329f 2023-07-12 16:08:17,863 INFO [M:0;jenkins-hbase5:39879] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9f7d80ed87754f249fcfa813ef8b329f 2023-07-12 16:08:17,864 INFO [M:0;jenkins-hbase5:39879] regionserver.HStore(1080): Added hdfs://localhost:34197/user/jenkins/test-data/d8d88999-7bde-2ff2-497d-81e64e1a14f9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9f7d80ed87754f249fcfa813ef8b329f, entries=166, sequenceid=1262, filesize=43.6 K 2023-07-12 16:08:17,864 INFO [M:0;jenkins-hbase5:39879] regionserver.HRegion(2948): Finished flush of dataSize ~486.68 KB/498360, heapSize ~589.59 KB/603744, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 439ms, sequenceid=1262, compaction requested=false 2023-07-12 16:08:17,866 INFO [M:0;jenkins-hbase5:39879] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 16:08:17,866 DEBUG [M:0;jenkins-hbase5:39879] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 16:08:17,872 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 16:08:17,872 INFO [M:0;jenkins-hbase5:39879] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 16:08:17,872 INFO [M:0;jenkins-hbase5:39879] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:39879 2023-07-12 16:08:17,874 DEBUG [M:0;jenkins-hbase5:39879] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,39879,1689177956989 already deleted, retry=false 2023-07-12 16:08:17,977 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 16:08:17,977 DEBUG [Listener at localhost/39041-EventThread] zookeeper.ZKWatcher(600): master:39879-0x101c5e4e98d0000, quorum=127.0.0.1:55421, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 16:08:17,977 INFO [M:0;jenkins-hbase5:39879] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,39879,1689177956989; zookeeper connection closed. 2023-07-12 16:08:17,979 WARN [Listener at localhost/39041] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 16:08:17,981 INFO [Listener at localhost/39041] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 16:08:18,087 WARN [BP-793531620-172.31.10.131-1689177953937 heartbeating to localhost/127.0.0.1:34197] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 16:08:18,087 WARN [BP-793531620-172.31.10.131-1689177953937 heartbeating to localhost/127.0.0.1:34197] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-793531620-172.31.10.131-1689177953937 (Datanode Uuid 6bcaae43-a6a2-463b-91b0-4540258288fe) service to localhost/127.0.0.1:34197 2023-07-12 16:08:18,089 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/cluster_fd62e4bd-2d7f-c592-2cc1-3edac70c12cd/dfs/data/data1/current/BP-793531620-172.31.10.131-1689177953937] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 16:08:18,089 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4d261ef2-e836-0be2-c752-56581a03866b/cluster_fd62e4bd-2d7f-c592-2cc1-3edac70c12cd/dfs/data/data2/current/BP-793531620-172.31.10.131-1689177953937] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 16:08:18,145 INFO [Listener at localhost/39041] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 16:08:18,256 INFO [Listener at localhost/39041] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 16:08:18,297 INFO [Listener at localhost/39041] hbase.HBaseTestingUtility(1293): Minicluster is down