2023-07-12 19:40:05,777 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.quotas.TestSnapshotQuotaObserverChore timeout: 13 mins 2023-07-12 19:40:05,996 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be 2023-07-12 19:40:06,012 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 19:40:06,012 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/cluster_cb8e40ba-4554-a1e8-3e84-9973d51e9321, deleteOnExit=true 2023-07-12 19:40:06,012 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 19:40:06,013 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/test.cache.data in system properties and HBase conf 2023-07-12 19:40:06,013 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 19:40:06,014 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/hadoop.log.dir in system properties and HBase conf 2023-07-12 19:40:06,014 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 19:40:06,014 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 19:40:06,015 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 19:40:06,133 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 19:40:06,502 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 19:40:06,506 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:40:06,506 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:40:06,507 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 19:40:06,507 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:40:06,507 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 19:40:06,507 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 19:40:06,508 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:40:06,508 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:40:06,509 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 19:40:06,509 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/nfs.dump.dir in system properties and HBase conf 2023-07-12 19:40:06,509 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/java.io.tmpdir in system properties and HBase conf 2023-07-12 19:40:06,510 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:40:06,510 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 19:40:06,511 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 19:40:06,978 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:40:06,982 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:40:07,276 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 19:40:07,424 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 19:40:07,439 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:40:07,476 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:40:07,537 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/java.io.tmpdir/Jetty_localhost_43645_hdfs____.5oqnzy/webapp 2023-07-12 19:40:07,669 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43645 2023-07-12 19:40:07,679 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:40:07,680 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:40:08,156 WARN [Listener at localhost/35215] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:40:08,223 WARN [Listener at localhost/35215] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 19:40:08,241 WARN [Listener at localhost/35215] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:40:08,247 INFO [Listener at localhost/35215] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:40:08,251 INFO [Listener at localhost/35215] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/java.io.tmpdir/Jetty_localhost_45685_datanode____td6j5l/webapp 2023-07-12 19:40:08,352 INFO [Listener at localhost/35215] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45685 2023-07-12 19:40:08,670 WARN [Listener at localhost/32953] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:40:09,087 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd602a81631b453b0: Processing first storage report for DS-42372720-5149-4f85-9bb2-a0e9fdbf8f74 from datanode 16e71f0f-a9d5-410c-b866-01f5fae0b1cd 2023-07-12 19:40:09,088 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd602a81631b453b0: from storage DS-42372720-5149-4f85-9bb2-a0e9fdbf8f74 node DatanodeRegistration(127.0.0.1:38967, datanodeUuid=16e71f0f-a9d5-410c-b866-01f5fae0b1cd, infoPort=33269, infoSecurePort=0, ipcPort=32953, storageInfo=lv=-57;cid=testClusterID;nsid=1091698012;c=1689190807059), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 19:40:09,089 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xd602a81631b453b0: Processing first storage report for DS-36084fe5-faec-48ea-bff0-f1edbf48e579 from datanode 16e71f0f-a9d5-410c-b866-01f5fae0b1cd 2023-07-12 19:40:09,089 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xd602a81631b453b0: from storage DS-36084fe5-faec-48ea-bff0-f1edbf48e579 node DatanodeRegistration(127.0.0.1:38967, datanodeUuid=16e71f0f-a9d5-410c-b866-01f5fae0b1cd, infoPort=33269, infoSecurePort=0, ipcPort=32953, storageInfo=lv=-57;cid=testClusterID;nsid=1091698012;c=1689190807059), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:40:09,163 DEBUG [Listener at localhost/32953] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be 2023-07-12 19:40:09,227 INFO [Listener at localhost/32953] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/cluster_cb8e40ba-4554-a1e8-3e84-9973d51e9321/zookeeper_0, clientPort=57973, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/cluster_cb8e40ba-4554-a1e8-3e84-9973d51e9321/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/cluster_cb8e40ba-4554-a1e8-3e84-9973d51e9321/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 19:40:09,240 INFO [Listener at localhost/32953] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57973 2023-07-12 19:40:09,251 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:40:09,254 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:40:09,929 INFO [Listener at localhost/32953] util.FSUtils(471): Created version file at hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151 with version=8 2023-07-12 19:40:09,930 INFO [Listener at localhost/32953] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/hbase-staging 2023-07-12 19:40:10,304 INFO [Listener at localhost/32953] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 19:40:10,781 INFO [Listener at localhost/32953] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:40:10,813 INFO [Listener at localhost/32953] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:40:10,814 INFO [Listener at localhost/32953] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:40:10,814 INFO [Listener at localhost/32953] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:40:10,814 INFO [Listener at localhost/32953] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:40:10,814 INFO [Listener at localhost/32953] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:40:10,973 INFO [Listener at localhost/32953] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:40:11,048 DEBUG [Listener at localhost/32953] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 19:40:11,156 INFO [Listener at localhost/32953] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33647 2023-07-12 19:40:11,171 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:40:11,174 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:40:11,204 INFO [Listener at localhost/32953] zookeeper.RecoverableZooKeeper(93): Process identifier=master:33647 connecting to ZooKeeper ensemble=127.0.0.1:57973 2023-07-12 19:40:11,248 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:336470x0, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:40:11,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:33647-0x1015b0cc17f0000 connected 2023-07-12 19:40:11,278 DEBUG [Listener at localhost/32953] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:40:11,279 DEBUG [Listener at localhost/32953] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:40:11,283 DEBUG [Listener at localhost/32953] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:40:11,293 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33647 2023-07-12 19:40:11,294 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33647 2023-07-12 19:40:11,294 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33647 2023-07-12 19:40:11,298 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33647 2023-07-12 19:40:11,299 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33647 2023-07-12 19:40:11,305 INFO [Listener at localhost/32953] master.HMaster(444): hbase.rootdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151, hbase.cluster.distributed=false 2023-07-12 19:40:11,374 INFO [Listener at localhost/32953] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:40:11,375 INFO [Listener at localhost/32953] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:40:11,375 INFO [Listener at localhost/32953] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:40:11,375 INFO [Listener at localhost/32953] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:40:11,375 INFO [Listener at localhost/32953] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:40:11,375 INFO [Listener at localhost/32953] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:40:11,380 INFO [Listener at localhost/32953] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:40:11,383 INFO [Listener at localhost/32953] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44631 2023-07-12 19:40:11,385 INFO [Listener at localhost/32953] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 19:40:11,392 DEBUG [Listener at localhost/32953] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 19:40:11,393 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:40:11,396 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:40:11,397 INFO [Listener at localhost/32953] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44631 connecting to ZooKeeper ensemble=127.0.0.1:57973 2023-07-12 19:40:11,401 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:446310x0, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:40:11,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:44631-0x1015b0cc17f0001 connected 2023-07-12 19:40:11,402 DEBUG [Listener at localhost/32953] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:40:11,404 DEBUG [Listener at localhost/32953] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:40:11,404 DEBUG [Listener at localhost/32953] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:40:11,405 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44631 2023-07-12 19:40:11,405 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44631 2023-07-12 19:40:11,406 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44631 2023-07-12 19:40:11,408 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44631 2023-07-12 19:40:11,409 DEBUG [Listener at localhost/32953] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44631 2023-07-12 19:40:11,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:40:11,419 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:40:11,420 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:40:11,439 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:40:11,439 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:40:11,439 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:40:11,440 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:40:11,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,33647,1689190810081 from backup master directory 2023-07-12 19:40:11,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:40:11,444 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:40:11,444 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:40:11,445 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:40:11,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:40:11,448 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 19:40:11,449 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 19:40:11,530 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/hbase.id with ID: fd859d27-d51f-4fac-9fac-cbb26209dded 2023-07-12 19:40:11,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:40:11,585 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:40:11,622 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x40530357 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:11,655 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@230e1181, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:11,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:11,679 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:40:11,697 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 19:40:11,697 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 19:40:11,699 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:11,703 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:11,704 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:40:11,741 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store-tmp 2023-07-12 19:40:12,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:12,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:40:12,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:40:12,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:40:12,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:40:12,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:40:12,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:40:12,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:40:12,170 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/WALs/jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:40:12,195 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33647%2C1689190810081, suffix=, logDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/WALs/jenkins-hbase4.apache.org,33647,1689190810081, archiveDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/oldWALs, maxLogs=10 2023-07-12 19:40:12,247 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38967,DS-42372720-5149-4f85-9bb2-a0e9fdbf8f74,DISK] 2023-07-12 19:40:12,254 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:12,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/WALs/jenkins-hbase4.apache.org,33647,1689190810081/jenkins-hbase4.apache.org%2C33647%2C1689190810081.1689190812205 2023-07-12 19:40:12,314 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38967,DS-42372720-5149-4f85-9bb2-a0e9fdbf8f74,DISK]] 2023-07-12 19:40:12,314 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:12,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:12,318 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:40:12,319 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:40:12,374 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:40:12,381 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:40:12,411 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:40:12,423 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:12,429 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:40:12,430 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:40:12,445 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:40:12,449 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:12,450 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61676152, jitterRate=-0.08095371723175049}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:12,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:40:12,451 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:40:12,470 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 19:40:12,471 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 19:40:12,473 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:40:12,475 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 19:40:12,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-07-12 19:40:12,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:40:12,534 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 19:40:12,539 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:40:12,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:40:12,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:40:12,579 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:40:12,586 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:40:12,592 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:40:12,596 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:40:12,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:40:12,598 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:40:12,614 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:40:12,620 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:40:12,620 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:40:12,620 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:40:12,621 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,33647,1689190810081, sessionid=0x1015b0cc17f0000, setting cluster-up flag (Was=false) 2023-07-12 19:40:12,635 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:40:12,641 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:40:12,642 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:40:12,648 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:40:12,653 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:40:12,654 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:40:12,657 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp 2023-07-12 19:40:12,699 INFO [master/jenkins-hbase4:0:becomeActiveMaster] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.quotas.MasterQuotasObserver loaded, priority=536870911. 2023-07-12 19:40:12,713 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(951): ClusterId : fd859d27-d51f-4fac-9fac-cbb26209dded 2023-07-12 19:40:12,717 DEBUG [RS:0;jenkins-hbase4:44631] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 19:40:12,724 DEBUG [RS:0;jenkins-hbase4:44631] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 19:40:12,724 DEBUG [RS:0;jenkins-hbase4:44631] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 19:40:12,727 DEBUG [RS:0;jenkins-hbase4:44631] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 19:40:12,729 DEBUG [RS:0;jenkins-hbase4:44631] zookeeper.ReadOnlyZKClient(139): Connect 0x131ff9bf to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:12,734 DEBUG [RS:0;jenkins-hbase4:44631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1dd815ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:12,734 DEBUG [RS:0;jenkins-hbase4:44631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e792cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:40:12,769 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:44631 2023-07-12 19:40:12,774 INFO [RS:0;jenkins-hbase4:44631] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 19:40:12,775 INFO [RS:0;jenkins-hbase4:44631] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 19:40:12,775 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 19:40:12,778 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 19:40:12,779 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,33647,1689190810081 with isa=jenkins-hbase4.apache.org/172.31.14.131:44631, startcode=1689190811374 2023-07-12 19:40:12,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:40:12,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:40:12,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:40:12,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:40:12,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:40:12,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:40:12,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,791 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689190842790 2023-07-12 19:40:12,793 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:40:12,797 DEBUG [RS:0;jenkins-hbase4:44631] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:40:12,797 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:40:12,798 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 19:40:12,803 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:12,806 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:40:12,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:40:12,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:40:12,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:40:12,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:40:12,817 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:40:12,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:40:12,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:40:12,824 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:40:12,824 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:40:12,827 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689190812826,5,FailOnTimeoutGroup] 2023-07-12 19:40:12,829 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689190812827,5,FailOnTimeoutGroup] 2023-07-12 19:40:12,829 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,829 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:40:12,831 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,831 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,857 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57693, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:40:12,867 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=33647] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:12,881 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151 2023-07-12 19:40:12,881 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:35215 2023-07-12 19:40:12,881 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 19:40:12,886 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:40:12,887 DEBUG [RS:0;jenkins-hbase4:44631] zookeeper.ZKUtil(162): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:12,887 WARN [RS:0;jenkins-hbase4:44631] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:40:12,888 INFO [RS:0;jenkins-hbase4:44631] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:40:12,888 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1948): logDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/WALs/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:12,889 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44631,1689190811374] 2023-07-12 19:40:12,896 DEBUG [RS:0;jenkins-hbase4:44631] zookeeper.ZKUtil(162): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:12,907 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 19:40:12,915 INFO [RS:0;jenkins-hbase4:44631] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 19:40:12,934 INFO [RS:0;jenkins-hbase4:44631] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 19:40:12,938 INFO [RS:0;jenkins-hbase4:44631] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:40:12,938 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,939 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 19:40:12,946 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=15000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,946 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,947 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,947 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,947 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,947 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,947 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:40:12,947 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,947 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,948 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,948 DEBUG [RS:0;jenkins-hbase4:44631] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:40:12,949 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,949 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,949 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,949 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=FileSystemUtilizationChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,965 INFO [RS:0;jenkins-hbase4:44631] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 19:40:12,967 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44631,1689190811374-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,983 INFO [RS:0;jenkins-hbase4:44631] regionserver.Replication(203): jenkins-hbase4.apache.org,44631,1689190811374 started 2023-07-12 19:40:12,983 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,44631,1689190811374, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44631, sessionid=0x1015b0cc17f0001 2023-07-12 19:40:12,984 DEBUG [RS:0;jenkins-hbase4:44631] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 19:40:12,984 DEBUG [RS:0;jenkins-hbase4:44631] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:12,984 DEBUG [RS:0;jenkins-hbase4:44631] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44631,1689190811374' 2023-07-12 19:40:12,984 DEBUG [RS:0;jenkins-hbase4:44631] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:12,984 DEBUG [RS:0;jenkins-hbase4:44631] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:12,985 DEBUG [RS:0;jenkins-hbase4:44631] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 19:40:12,985 DEBUG [RS:0;jenkins-hbase4:44631] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 19:40:12,985 DEBUG [RS:0;jenkins-hbase4:44631] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:12,985 DEBUG [RS:0;jenkins-hbase4:44631] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44631,1689190811374' 2023-07-12 19:40:12,985 DEBUG [RS:0;jenkins-hbase4:44631] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:12,985 DEBUG [RS:0;jenkins-hbase4:44631] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:12,986 DEBUG [RS:0;jenkins-hbase4:44631] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 19:40:12,986 INFO [RS:0;jenkins-hbase4:44631] quotas.RegionServerRpcQuotaManager(67): Initializing RPC quota support 2023-07-12 19:40:12,988 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=QuotaRefresherChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,988 DEBUG [RS:0;jenkins-hbase4:44631] zookeeper.ZKUtil(398): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/rpc-throttle because node does not exist (not an error) 2023-07-12 19:40:12,988 INFO [RS:0;jenkins-hbase4:44631] quotas.RegionServerRpcQuotaManager(73): Start rpc quota manager and rpc throttle enabled is true 2023-07-12 19:40:12,989 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=SpaceQuotaRefresherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:12,989 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(166): Chore ScheduledChore name=RegionSizeReportingChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:13,103 INFO [RS:0;jenkins-hbase4:44631] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44631%2C1689190811374, suffix=, logDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/WALs/jenkins-hbase4.apache.org,44631,1689190811374, archiveDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/oldWALs, maxLogs=32 2023-07-12 19:40:13,128 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38967,DS-42372720-5149-4f85-9bb2-a0e9fdbf8f74,DISK] 2023-07-12 19:40:13,131 INFO [RS:0;jenkins-hbase4:44631] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/WALs/jenkins-hbase4.apache.org,44631,1689190811374/jenkins-hbase4.apache.org%2C44631%2C1689190811374.1689190813107 2023-07-12 19:40:13,131 DEBUG [RS:0;jenkins-hbase4:44631] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38967,DS-42372720-5149-4f85-9bb2-a0e9fdbf8f74,DISK]] 2023-07-12 19:40:13,243 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:13,244 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:13,244 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151 2023-07-12 19:40:13,659 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:13,662 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:40:13,665 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/info 2023-07-12 19:40:13,666 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:40:13,667 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:13,667 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:40:13,670 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:40:13,670 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:40:13,671 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:13,671 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:40:13,673 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/table 2023-07-12 19:40:13,674 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:40:13,675 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:13,676 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740 2023-07-12 19:40:13,677 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740 2023-07-12 19:40:13,680 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:40:13,682 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:40:13,686 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:13,686 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61457118, jitterRate=-0.08421757817268372}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:40:13,687 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:40:13,687 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:40:13,687 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:40:13,687 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:40:13,687 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:40:13,687 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:40:13,688 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:40:13,688 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:40:13,692 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:40:13,692 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 19:40:13,700 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 19:40:13,711 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 19:40:13,713 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 19:40:13,865 DEBUG [jenkins-hbase4:33647] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 19:40:13,868 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44631,1689190811374, state=OPENING 2023-07-12 19:40:13,877 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 19:40:13,878 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:40:13,879 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:40:13,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:14,066 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:14,069 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:40:14,073 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:40:14,088 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 19:40:14,089 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:40:14,092 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44631%2C1689190811374.meta, suffix=.meta, logDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/WALs/jenkins-hbase4.apache.org,44631,1689190811374, archiveDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/oldWALs, maxLogs=32 2023-07-12 19:40:14,110 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38967,DS-42372720-5149-4f85-9bb2-a0e9fdbf8f74,DISK] 2023-07-12 19:40:14,117 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/WALs/jenkins-hbase4.apache.org,44631,1689190811374/jenkins-hbase4.apache.org%2C44631%2C1689190811374.meta.1689190814093.meta 2023-07-12 19:40:14,122 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38967,DS-42372720-5149-4f85-9bb2-a0e9fdbf8f74,DISK]] 2023-07-12 19:40:14,123 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:14,125 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 19:40:14,144 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 19:40:14,148 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 19:40:14,153 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 19:40:14,153 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:14,153 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 19:40:14,153 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 19:40:14,156 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:40:14,158 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/info 2023-07-12 19:40:14,158 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/info 2023-07-12 19:40:14,159 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:40:14,159 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:14,160 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:40:14,161 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:40:14,161 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:40:14,162 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:40:14,163 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:14,163 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:40:14,164 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/table 2023-07-12 19:40:14,164 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/table 2023-07-12 19:40:14,165 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:40:14,166 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:14,167 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740 2023-07-12 19:40:14,170 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740 2023-07-12 19:40:14,174 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:40:14,176 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:40:14,177 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59773875, jitterRate=-0.10929985344409943}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:40:14,178 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:40:14,187 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689190814058 2023-07-12 19:40:14,204 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 19:40:14,204 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 19:40:14,205 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44631,1689190811374, state=OPEN 2023-07-12 19:40:14,207 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:40:14,207 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:40:14,211 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 19:40:14,211 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44631,1689190811374 in 325 msec 2023-07-12 19:40:14,216 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 19:40:14,216 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 513 msec 2023-07-12 19:40:14,221 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5140 sec 2023-07-12 19:40:14,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689190814221, completionTime=-1 2023-07-12 19:40:14,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:40:14,222 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:40:14,286 DEBUG [hconnection-0x3e827ec6-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:14,290 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43822, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:14,308 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:40:14,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689190874309 2023-07-12 19:40:14,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689190934309 2023-07-12 19:40:14,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 86 msec 2023-07-12 19:40:14,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33647,1689190810081-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:14,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33647,1689190810081-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:14,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33647,1689190810081-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:14,336 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:33647, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:14,336 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:14,342 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:40:14,350 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 19:40:14,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:14,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 19:40:14,364 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:14,368 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:14,395 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:14,397 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7 empty. 2023-07-12 19:40:14,398 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:14,399 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 19:40:14,824 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:14,827 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 997e5e40c84648e410710e2c371a54e7, NAME => 'hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:14,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2840) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionSpaceUse(MasterRpcServices.java:2272) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15962) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:40:14,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.CallRunner(144): callId: 20 service: RegionServerStatusService methodName: ReportRegionSpaceUse size: 66 connection: 172.31.14.131:57693 deadline: 1689190824995, exception=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing 2023-07-12 19:40:15,026 INFO [regionserver/jenkins-hbase4:0.Chore.1] quotas.SpaceQuotaRefresherChore(80): Quota table not found, skipping quota manager cache refresh. 2023-07-12 19:40:15,239 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:15,239 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 997e5e40c84648e410710e2c371a54e7, disabling compactions & flushes 2023-07-12 19:40:15,239 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:40:15,239 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:40:15,239 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. after waiting 0 ms 2023-07-12 19:40:15,239 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:40:15,239 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:40:15,239 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 997e5e40c84648e410710e2c371a54e7: 2023-07-12 19:40:15,245 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:15,261 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689190815247"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190815247"}]},"ts":"1689190815247"} 2023-07-12 19:40:15,290 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:15,292 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:15,296 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190815292"}]},"ts":"1689190815292"} 2023-07-12 19:40:15,301 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 19:40:15,310 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=997e5e40c84648e410710e2c371a54e7, ASSIGN}] 2023-07-12 19:40:15,312 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=997e5e40c84648e410710e2c371a54e7, ASSIGN 2023-07-12 19:40:15,314 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=997e5e40c84648e410710e2c371a54e7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:15,465 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=997e5e40c84648e410710e2c371a54e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:15,466 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689190815465"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190815465"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190815465"}]},"ts":"1689190815465"} 2023-07-12 19:40:15,471 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 997e5e40c84648e410710e2c371a54e7, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:15,635 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:40:15,636 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 997e5e40c84648e410710e2c371a54e7, NAME => 'hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:15,637 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:15,637 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:15,637 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:15,637 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:15,640 INFO [StoreOpener-997e5e40c84648e410710e2c371a54e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:15,644 DEBUG [StoreOpener-997e5e40c84648e410710e2c371a54e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7/info 2023-07-12 19:40:15,645 DEBUG [StoreOpener-997e5e40c84648e410710e2c371a54e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7/info 2023-07-12 19:40:15,645 INFO [StoreOpener-997e5e40c84648e410710e2c371a54e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 997e5e40c84648e410710e2c371a54e7 columnFamilyName info 2023-07-12 19:40:15,646 INFO [StoreOpener-997e5e40c84648e410710e2c371a54e7-1] regionserver.HStore(310): Store=997e5e40c84648e410710e2c371a54e7/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:15,648 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:15,650 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:15,658 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 997e5e40c84648e410710e2c371a54e7 2023-07-12 19:40:15,663 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:15,664 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 997e5e40c84648e410710e2c371a54e7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72131872, jitterRate=0.07484865188598633}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:15,665 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 997e5e40c84648e410710e2c371a54e7: 2023-07-12 19:40:15,667 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7., pid=6, masterSystemTime=1689190815626 2023-07-12 19:40:15,672 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:40:15,672 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:40:15,674 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=997e5e40c84648e410710e2c371a54e7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:15,675 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689190815673"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190815673"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190815673"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190815673"}]},"ts":"1689190815673"} 2023-07-12 19:40:15,683 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 19:40:15,683 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 997e5e40c84648e410710e2c371a54e7, server=jenkins-hbase4.apache.org,44631,1689190811374 in 208 msec 2023-07-12 19:40:15,687 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 19:40:15,688 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=997e5e40c84648e410710e2c371a54e7, ASSIGN in 374 msec 2023-07-12 19:40:15,689 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:15,689 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190815689"}]},"ts":"1689190815689"} 2023-07-12 19:40:15,695 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 19:40:15,701 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:15,705 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3490 sec 2023-07-12 19:40:15,768 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 19:40:15,770 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:40:15,770 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:40:15,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 19:40:15,834 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:40:15,841 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-07-12 19:40:15,850 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:40:15,868 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:40:15,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 23 msec 2023-07-12 19:40:15,886 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:40:15,889 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:40:15,889 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.444sec 2023-07-12 19:40:15,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(103): Quota table not found. Creating... 2023-07-12 19:40:15,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:quota', {NAME => 'q', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'u', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:15,893 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:quota 2023-07-12 19:40:15,893 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(107): Initializing quota support 2023-07-12 19:40:15,895 INFO [master/jenkins-hbase4:0:becomeActiveMaster] namespace.NamespaceStateManager(59): Namespace State Manager started. 2023-07-12 19:40:15,895 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:15,897 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:15,900 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:15,901 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209 empty. 2023-07-12 19:40:15,903 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:15,903 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:quota regions 2023-07-12 19:40:15,904 INFO [master/jenkins-hbase4:0:becomeActiveMaster] namespace.NamespaceStateManager(222): Finished updating state of 2 namespaces. 2023-07-12 19:40:15,904 INFO [master/jenkins-hbase4:0:becomeActiveMaster] namespace.NamespaceAuditor(50): NamespaceAuditor started. 2023-07-12 19:40:15,906 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=QuotaObserverChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:15,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=QuotaObserverChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:15,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:40:15,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:40:15,910 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33647,1689190810081-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:40:15,911 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33647,1689190810081-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:40:15,923 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x309d5bdc to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:15,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:40:15,928 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c9912ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:15,945 DEBUG [hconnection-0x676e705e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:15,957 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43828, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:15,960 INFO [Listener at localhost/32953] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:40:15,982 INFO [Listener at localhost/32953] hbase.ResourceChecker(147): before: quotas.TestSnapshotQuotaObserverChore#testSnapshotsFromTables Thread=212, OpenFileDescriptor=473, MaxFileDescriptor=60000, SystemLoadAverage=124, ProcessCount=172, AvailableMemoryMB=8210 2023-07-12 19:40:15,989 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:40:15,993 INFO [regionserver/jenkins-hbase4:0.Chore.1] quotas.SpaceQuotaRefresherChore(80): Quota table not found, skipping quota manager cache refresh. 2023-07-12 19:40:16,323 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/hbase/quota/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:16,325 INFO [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(7675): creating {ENCODED => 05380a24dc8e79f5b99fe9c98393c209, NAME => 'hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:quota', {NAME => 'q', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'u', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:16,738 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(866): Instantiated hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:16,738 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1604): Closing 05380a24dc8e79f5b99fe9c98393c209, disabling compactions & flushes 2023-07-12 19:40:16,738 INFO [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1626): Closing region hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:40:16,738 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:40:16,738 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. after waiting 0 ms 2023-07-12 19:40:16,738 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:40:16,738 INFO [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1838): Closed hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:40:16,739 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1558): Region close journal for 05380a24dc8e79f5b99fe9c98393c209: 2023-07-12 19:40:16,743 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:16,745 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209.","families":{"info":[{"qualifier":"regioninfo","vlen":37,"tag":[],"timestamp":"1689190816745"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190816745"}]},"ts":"1689190816745"} 2023-07-12 19:40:16,748 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:16,749 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:16,750 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:quota","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190816750"}]},"ts":"1689190816750"} 2023-07-12 19:40:16,752 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:quota, state=ENABLING in hbase:meta 2023-07-12 19:40:16,758 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:quota, region=05380a24dc8e79f5b99fe9c98393c209, ASSIGN}] 2023-07-12 19:40:16,760 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:quota, region=05380a24dc8e79f5b99fe9c98393c209, ASSIGN 2023-07-12 19:40:16,762 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:quota, region=05380a24dc8e79f5b99fe9c98393c209, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:16,913 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=05380a24dc8e79f5b99fe9c98393c209, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:16,914 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209.","families":{"info":[{"qualifier":"regioninfo","vlen":37,"tag":[],"timestamp":"1689190816913"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190816913"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190816913"}]},"ts":"1689190816913"} 2023-07-12 19:40:16,917 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 05380a24dc8e79f5b99fe9c98393c209, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:16,997 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] client.ConnectionImplementation(1060): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 46 failed; retrying after sleep of 46 org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in hbase:meta for region hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. containing row t. at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:1029) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:866) at org.apache.hadoop.hbase.client.ServerConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ServerConnectionUtils.java:70) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:157) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:266) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:309) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:594) at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:51) at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:186) at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:91) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:17,002 DEBUG [Listener at localhost/32953] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 19:40:17,006 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 19:40:17,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testSnapshotsFromTables0', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:17,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSnapshotsFromTables0 2023-07-12 19:40:17,021 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSnapshotsFromTables0 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:17,023 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSnapshotsFromTables0 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:17,025 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:17,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testSnapshotsFromTables0" procId is: 12 2023-07-12 19:40:17,027 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0 empty. 2023-07-12 19:40:17,028 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:17,028 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSnapshotsFromTables0 regions 2023-07-12 19:40:17,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:40:17,075 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:40:17,076 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 05380a24dc8e79f5b99fe9c98393c209, NAME => 'hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:17,076 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table quota 05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:17,076 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:17,076 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:17,076 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:17,078 INFO [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family q of region 05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:17,081 DEBUG [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/q 2023-07-12 19:40:17,081 DEBUG [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/q 2023-07-12 19:40:17,081 INFO [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 05380a24dc8e79f5b99fe9c98393c209 columnFamilyName q 2023-07-12 19:40:17,082 INFO [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] regionserver.HStore(310): Store=05380a24dc8e79f5b99fe9c98393c209/q, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:17,083 INFO [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family u of region 05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:17,085 DEBUG [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/u 2023-07-12 19:40:17,085 DEBUG [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/u 2023-07-12 19:40:17,085 INFO [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 05380a24dc8e79f5b99fe9c98393c209 columnFamilyName u 2023-07-12 19:40:17,086 INFO [StoreOpener-05380a24dc8e79f5b99fe9c98393c209-1] regionserver.HStore(310): Store=05380a24dc8e79f5b99fe9c98393c209/u, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:17,088 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:17,089 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:17,092 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:quota descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:40:17,095 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:40:17,101 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] client.ConnectionImplementation(1060): locateRegionInMeta parentTable='hbase:meta', attempt=1 of 46 failed; retrying after sleep of 46 org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in hbase:meta for region hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. containing row t. at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:1029) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:866) at org.apache.hadoop.hbase.client.ServerConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ServerConnectionUtils.java:70) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:157) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:266) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:309) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:594) at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:51) at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:186) at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:91) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:17,110 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:17,111 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 05380a24dc8e79f5b99fe9c98393c209; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73217500, jitterRate=0.09102576971054077}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:40:17,111 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 05380a24dc8e79f5b99fe9c98393c209: 2023-07-12 19:40:17,113 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209., pid=11, masterSystemTime=1689190817070 2023-07-12 19:40:17,115 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:40:17,115 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:40:17,116 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=05380a24dc8e79f5b99fe9c98393c209, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:17,117 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209.","families":{"info":[{"qualifier":"regioninfo","vlen":37,"tag":[],"timestamp":"1689190817116"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190817116"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190817116"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190817116"}]},"ts":"1689190817116"} 2023-07-12 19:40:17,123 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 19:40:17,123 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 05380a24dc8e79f5b99fe9c98393c209, server=jenkins-hbase4.apache.org,44631,1689190811374 in 202 msec 2023-07-12 19:40:17,127 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 19:40:17,127 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=hbase:quota, region=05380a24dc8e79f5b99fe9c98393c209, ASSIGN in 365 msec 2023-07-12 19:40:17,128 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:17,128 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:quota","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190817128"}]},"ts":"1689190817128"} 2023-07-12 19:40:17,130 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:quota, state=ENABLED in hbase:meta 2023-07-12 19:40:17,137 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:17,139 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=hbase:quota in 1.2460 sec 2023-07-12 19:40:17,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:40:17,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:40:17,447 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables0/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:17,448 INFO [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(7675): creating {ENCODED => cabde14ae3b55e4b2cc7f9e1b1f244b0, NAME => 'testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSnapshotsFromTables0', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:17,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:40:17,861 DEBUG [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(866): Instantiated testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:17,862 DEBUG [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(1604): Closing cabde14ae3b55e4b2cc7f9e1b1f244b0, disabling compactions & flushes 2023-07-12 19:40:17,862 INFO [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(1626): Closing region testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:17,862 DEBUG [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:17,862 DEBUG [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. after waiting 0 ms 2023-07-12 19:40:17,862 DEBUG [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:17,862 INFO [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(1838): Closed testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:17,862 DEBUG [RegionOpenAndInit-testSnapshotsFromTables0-pool-0] regionserver.HRegion(1558): Region close journal for cabde14ae3b55e4b2cc7f9e1b1f244b0: 2023-07-12 19:40:17,866 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSnapshotsFromTables0 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:17,868 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190817868"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190817868"}]},"ts":"1689190817868"} 2023-07-12 19:40:17,870 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:17,872 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSnapshotsFromTables0 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:17,872 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotsFromTables0","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190817872"}]},"ts":"1689190817872"} 2023-07-12 19:40:17,874 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotsFromTables0, state=ENABLING in hbase:meta 2023-07-12 19:40:17,878 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotsFromTables0, region=cabde14ae3b55e4b2cc7f9e1b1f244b0, ASSIGN}] 2023-07-12 19:40:17,880 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotsFromTables0, region=cabde14ae3b55e4b2cc7f9e1b1f244b0, ASSIGN 2023-07-12 19:40:17,881 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotsFromTables0, region=cabde14ae3b55e4b2cc7f9e1b1f244b0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:17,921 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2e72abd6 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:17,927 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3269e5b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:17,929 DEBUG [hconnection-0x424f2f0c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:17,930 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:18,032 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cabde14ae3b55e4b2cc7f9e1b1f244b0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:18,032 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190818032"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190818032"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190818032"}]},"ts":"1689190818032"} 2023-07-12 19:40:18,035 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure cabde14ae3b55e4b2cc7f9e1b1f244b0, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:18,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:40:18,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:18,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cabde14ae3b55e4b2cc7f9e1b1f244b0, NAME => 'testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:18,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotsFromTables0 cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:18,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:18,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:18,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:18,197 INFO [StoreOpener-cabde14ae3b55e4b2cc7f9e1b1f244b0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:18,199 DEBUG [StoreOpener-cabde14ae3b55e4b2cc7f9e1b1f244b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0/f1 2023-07-12 19:40:18,200 DEBUG [StoreOpener-cabde14ae3b55e4b2cc7f9e1b1f244b0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0/f1 2023-07-12 19:40:18,200 INFO [StoreOpener-cabde14ae3b55e4b2cc7f9e1b1f244b0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cabde14ae3b55e4b2cc7f9e1b1f244b0 columnFamilyName f1 2023-07-12 19:40:18,201 INFO [StoreOpener-cabde14ae3b55e4b2cc7f9e1b1f244b0-1] regionserver.HStore(310): Store=cabde14ae3b55e4b2cc7f9e1b1f244b0/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:18,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:18,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:18,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:40:18,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:18,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cabde14ae3b55e4b2cc7f9e1b1f244b0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62584716, jitterRate=-0.06741505861282349}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:18,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cabde14ae3b55e4b2cc7f9e1b1f244b0: 2023-07-12 19:40:18,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0., pid=14, masterSystemTime=1689190818188 2023-07-12 19:40:18,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:18,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:18,220 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cabde14ae3b55e4b2cc7f9e1b1f244b0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:18,220 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190818220"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190818220"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190818220"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190818220"}]},"ts":"1689190818220"} 2023-07-12 19:40:18,226 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 19:40:18,226 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure cabde14ae3b55e4b2cc7f9e1b1f244b0, server=jenkins-hbase4.apache.org,44631,1689190811374 in 188 msec 2023-07-12 19:40:18,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-12 19:40:18,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotsFromTables0, region=cabde14ae3b55e4b2cc7f9e1b1f244b0, ASSIGN in 348 msec 2023-07-12 19:40:18,230 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSnapshotsFromTables0 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:18,231 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotsFromTables0","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190818231"}]},"ts":"1689190818231"} 2023-07-12 19:40:18,233 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotsFromTables0, state=ENABLED in hbase:meta 2023-07-12 19:40:18,236 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=12, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSnapshotsFromTables0 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:18,238 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; CreateTableProcedure table=testSnapshotsFromTables0 in 1.2180 sec 2023-07-12 19:40:18,910 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2985a8e8 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:18,914 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 19:40:18,915 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3341cc38, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:18,916 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 19:40:18,917 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSnapshotsFromTables0' 2023-07-12 19:40:18,917 DEBUG [hconnection-0x761e6d15-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:18,918 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:quota' 2023-07-12 19:40:18,920 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43844, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:19,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:40:19,152 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testSnapshotsFromTables0, procId: 12 completed 2023-07-12 19:40:19,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testSnapshotsFromTables1', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:19,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSnapshotsFromTables1 2023-07-12 19:40:19,160 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSnapshotsFromTables1 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:19,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testSnapshotsFromTables1" procId is: 15 2023-07-12 19:40:19,162 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSnapshotsFromTables1 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:19,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=15 2023-07-12 19:40:19,169 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,170 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55 empty. 2023-07-12 19:40:19,171 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,171 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testSnapshotsFromTables1 regions 2023-07-12 19:40:19,215 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e72abd6 to 127.0.0.1:57973 2023-07-12 19:40:19,216 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:19,220 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2985a8e8 to 127.0.0.1:57973 2023-07-12 19:40:19,220 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:19,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=15 2023-07-12 19:40:19,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=15 2023-07-12 19:40:19,621 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables1/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:19,623 INFO [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3ed8dd7a6148cfa9771a9de1748ddf55, NAME => 'testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSnapshotsFromTables1', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:19,635 DEBUG [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(866): Instantiated testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:19,635 DEBUG [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(1604): Closing 3ed8dd7a6148cfa9771a9de1748ddf55, disabling compactions & flushes 2023-07-12 19:40:19,635 INFO [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(1626): Closing region testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:19,635 DEBUG [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:19,635 DEBUG [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. after waiting 0 ms 2023-07-12 19:40:19,635 DEBUG [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:19,635 INFO [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(1838): Closed testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:19,635 DEBUG [RegionOpenAndInit-testSnapshotsFromTables1-pool-0] regionserver.HRegion(1558): Region close journal for 3ed8dd7a6148cfa9771a9de1748ddf55: 2023-07-12 19:40:19,639 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSnapshotsFromTables1 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:19,640 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190819640"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190819640"}]},"ts":"1689190819640"} 2023-07-12 19:40:19,642 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:19,643 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSnapshotsFromTables1 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:19,644 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotsFromTables1","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190819643"}]},"ts":"1689190819643"} 2023-07-12 19:40:19,645 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotsFromTables1, state=ENABLING in hbase:meta 2023-07-12 19:40:19,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotsFromTables1, region=3ed8dd7a6148cfa9771a9de1748ddf55, ASSIGN}] 2023-07-12 19:40:19,651 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotsFromTables1, region=3ed8dd7a6148cfa9771a9de1748ddf55, ASSIGN 2023-07-12 19:40:19,652 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=15, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotsFromTables1, region=3ed8dd7a6148cfa9771a9de1748ddf55, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:19,679 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 19:40:19,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=15 2023-07-12 19:40:19,802 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=3ed8dd7a6148cfa9771a9de1748ddf55, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:19,803 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190819802"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190819802"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190819802"}]},"ts":"1689190819802"} 2023-07-12 19:40:19,805 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 3ed8dd7a6148cfa9771a9de1748ddf55, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:19,911 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1a46a97a to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:19,916 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10f41cd5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:19,918 DEBUG [hconnection-0x7a3282d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:19,920 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43858, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:19,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:19,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3ed8dd7a6148cfa9771a9de1748ddf55, NAME => 'testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:19,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotsFromTables1 3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:19,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,965 INFO [StoreOpener-3ed8dd7a6148cfa9771a9de1748ddf55-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,968 DEBUG [StoreOpener-3ed8dd7a6148cfa9771a9de1748ddf55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55/f1 2023-07-12 19:40:19,968 DEBUG [StoreOpener-3ed8dd7a6148cfa9771a9de1748ddf55-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55/f1 2023-07-12 19:40:19,968 INFO [StoreOpener-3ed8dd7a6148cfa9771a9de1748ddf55-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3ed8dd7a6148cfa9771a9de1748ddf55 columnFamilyName f1 2023-07-12 19:40:19,969 INFO [StoreOpener-3ed8dd7a6148cfa9771a9de1748ddf55-1] regionserver.HStore(310): Store=3ed8dd7a6148cfa9771a9de1748ddf55/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:19,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:40:19,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:19,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3ed8dd7a6148cfa9771a9de1748ddf55; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67190021, jitterRate=0.0012093335390090942}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:19,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3ed8dd7a6148cfa9771a9de1748ddf55: 2023-07-12 19:40:19,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55., pid=17, masterSystemTime=1689190819958 2023-07-12 19:40:19,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:19,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:19,986 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=3ed8dd7a6148cfa9771a9de1748ddf55, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:19,986 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190819986"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190819986"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190819986"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190819986"}]},"ts":"1689190819986"} 2023-07-12 19:40:19,991 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-07-12 19:40:19,991 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 3ed8dd7a6148cfa9771a9de1748ddf55, server=jenkins-hbase4.apache.org,44631,1689190811374 in 183 msec 2023-07-12 19:40:19,994 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-12 19:40:19,995 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotsFromTables1, region=3ed8dd7a6148cfa9771a9de1748ddf55, ASSIGN in 342 msec 2023-07-12 19:40:19,995 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSnapshotsFromTables1 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:19,996 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotsFromTables1","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190819996"}]},"ts":"1689190819996"} 2023-07-12 19:40:19,997 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotsFromTables1, state=ENABLED in hbase:meta 2023-07-12 19:40:20,002 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=15, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSnapshotsFromTables1 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:20,003 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; CreateTableProcedure table=testSnapshotsFromTables1 in 846 msec 2023-07-12 19:40:20,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=15 2023-07-12 19:40:20,270 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testSnapshotsFromTables1, procId: 15 completed 2023-07-12 19:40:20,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testSnapshotsFromTables2', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:20,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSnapshotsFromTables2 2023-07-12 19:40:20,276 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSnapshotsFromTables2 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:20,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testSnapshotsFromTables2" procId is: 18 2023-07-12 19:40:20,277 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSnapshotsFromTables2 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:20,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:40:20,279 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:20,280 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac empty. 2023-07-12 19:40:20,281 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:20,281 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testSnapshotsFromTables2 regions 2023-07-12 19:40:20,295 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.quotas.MasterQuotasObserver 2023-07-12 19:40:20,296 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.quotas.MasterQuotasObserver Metrics about HBase MasterObservers 2023-07-12 19:40:20,296 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:40:20,296 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 19:40:20,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:40:20,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:40:20,697 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromTables2/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:20,698 INFO [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9014d82a1855eaedeb89f1edd3e032ac, NAME => 'testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSnapshotsFromTables2', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:20,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:40:20,912 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1d23dcc6 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:20,919 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b0fde88, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:20,921 DEBUG [hconnection-0x73201998-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:20,923 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34946, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:21,113 DEBUG [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(866): Instantiated testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:21,114 DEBUG [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(1604): Closing 9014d82a1855eaedeb89f1edd3e032ac, disabling compactions & flushes 2023-07-12 19:40:21,114 INFO [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(1626): Closing region testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:21,114 DEBUG [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:21,114 DEBUG [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. after waiting 0 ms 2023-07-12 19:40:21,114 DEBUG [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:21,114 INFO [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(1838): Closed testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:21,114 DEBUG [RegionOpenAndInit-testSnapshotsFromTables2-pool-0] regionserver.HRegion(1558): Region close journal for 9014d82a1855eaedeb89f1edd3e032ac: 2023-07-12 19:40:21,118 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSnapshotsFromTables2 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:21,120 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190821120"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190821120"}]},"ts":"1689190821120"} 2023-07-12 19:40:21,123 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:21,124 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSnapshotsFromTables2 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:21,125 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotsFromTables2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190821125"}]},"ts":"1689190821125"} 2023-07-12 19:40:21,127 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotsFromTables2, state=ENABLING in hbase:meta 2023-07-12 19:40:21,131 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotsFromTables2, region=9014d82a1855eaedeb89f1edd3e032ac, ASSIGN}] 2023-07-12 19:40:21,134 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotsFromTables2, region=9014d82a1855eaedeb89f1edd3e032ac, ASSIGN 2023-07-12 19:40:21,135 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotsFromTables2, region=9014d82a1855eaedeb89f1edd3e032ac, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:21,286 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=9014d82a1855eaedeb89f1edd3e032ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:21,286 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190821286"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190821286"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190821286"}]},"ts":"1689190821286"} 2023-07-12 19:40:21,289 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE; OpenRegionProcedure 9014d82a1855eaedeb89f1edd3e032ac, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:40:21,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:21,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9014d82a1855eaedeb89f1edd3e032ac, NAME => 'testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:21,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotsFromTables2 9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:21,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:21,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:21,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:21,449 INFO [StoreOpener-9014d82a1855eaedeb89f1edd3e032ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:21,451 DEBUG [StoreOpener-9014d82a1855eaedeb89f1edd3e032ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac/f1 2023-07-12 19:40:21,451 DEBUG [StoreOpener-9014d82a1855eaedeb89f1edd3e032ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac/f1 2023-07-12 19:40:21,451 INFO [StoreOpener-9014d82a1855eaedeb89f1edd3e032ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9014d82a1855eaedeb89f1edd3e032ac columnFamilyName f1 2023-07-12 19:40:21,452 INFO [StoreOpener-9014d82a1855eaedeb89f1edd3e032ac-1] regionserver.HStore(310): Store=9014d82a1855eaedeb89f1edd3e032ac/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:21,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:21,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:21,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:40:21,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:21,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9014d82a1855eaedeb89f1edd3e032ac; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69270753, jitterRate=0.032214656472206116}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:21,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9014d82a1855eaedeb89f1edd3e032ac: 2023-07-12 19:40:21,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac., pid=20, masterSystemTime=1689190821441 2023-07-12 19:40:21,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:21,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:21,466 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=9014d82a1855eaedeb89f1edd3e032ac, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:21,466 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":"1689190821466"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190821466"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190821466"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190821466"}]},"ts":"1689190821466"} 2023-07-12 19:40:21,471 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-12 19:40:21,472 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; OpenRegionProcedure 9014d82a1855eaedeb89f1edd3e032ac, server=jenkins-hbase4.apache.org,44631,1689190811374 in 180 msec 2023-07-12 19:40:21,474 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-12 19:40:21,475 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotsFromTables2, region=9014d82a1855eaedeb89f1edd3e032ac, ASSIGN in 341 msec 2023-07-12 19:40:21,476 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSnapshotsFromTables2 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:21,476 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotsFromTables2","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190821476"}]},"ts":"1689190821476"} 2023-07-12 19:40:21,478 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotsFromTables2, state=ENABLED in hbase:meta 2023-07-12 19:40:21,482 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=18, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSnapshotsFromTables2 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:21,484 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; CreateTableProcedure table=testSnapshotsFromTables2 in 1.2090 sec 2023-07-12 19:40:21,909 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x36499595 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:21,914 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@231d716e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:21,916 DEBUG [hconnection-0x37f849-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:21,917 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:22,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:40:22,387 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testSnapshotsFromTables2, procId: 18 completed 2023-07-12 19:40:22,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:22,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190822427 (current time:1689190822427). 2023-07-12 19:40:22,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:22,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x7f36f724 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:22,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21dbc130, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:22,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:22,437 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:22,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f36f724 to 127.0.0.1:57973 2023-07-12 19:40:22,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:22,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:22,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=EXCLUSIVE 2023-07-12 19:40:22,457 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=21, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=EXCLUSIVE 2023-07-12 19:40:22,458 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=21, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=EXCLUSIVE; timeout=600000, timestamp=1689191422458 2023-07-12 19:40:22,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:22,458 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromTables0snapshot C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables0 2023-07-12 19:40:22,460 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=21, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=EXCLUSIVE 2023-07-12 19:40:22,461 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=SHARED 2023-07-12 19:40:22,462 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=EXCLUSIVE in 11 msec 2023-07-12 19:40:22,462 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=22, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=SHARED 2023-07-12 19:40:22,463 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:22,463 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:22,463 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=22, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=SHARED; timeout=600000, timestamp=1689191422463 2023-07-12 19:40:22,563 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:22,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:22,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:22,568 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:22,768 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:22,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:22,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:22,770 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:22,883 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,884 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromTables0snapshot' 2023-07-12 19:40:22,884 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:22,884 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromTables0snapshot' starting 'acquire' 2023-07-12 19:40:22,884 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromTables0snapshot', kicking off acquire phase on members. 2023-07-12 19:40:22,885 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,885 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,887 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:22,887 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:22,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:22,888 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,888 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:22,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 77 2023-07-12 19:40:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,889 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromTables0snapshot from table testSnapshotsFromTables0 type SKIPFLUSH 2023-07-12 19:40:22,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromTables0snapshot' with timeout 300000ms 2023-07-12 19:40:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:22,892 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromTables0snapshot' starting 'acquire' stage 2023-07-12 19:40:22,892 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromTables0snapshot' locally acquired 2023-07-12 19:40:22,892 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromTables0snapshot) in zk 2023-07-12 19:40:22,895 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:22,895 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:22,896 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,896 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromTables0snapshot' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:22,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromTables0snapshot' on coordinator 2023-07-12 19:40:22,899 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromTables0snapshot' starting 'in-barrier' execution. 2023-07-12 19:40:22,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e13a79b[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:22,899 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,901 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,901 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromTables0snapshot' received 'reached' from coordinator. 2023-07-12 19:40:22,901 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:22,901 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:22,902 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:22,902 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:22,903 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0.' region-info for snapshot=testSnapshotsFromTables0snapshot 2023-07-12 19:40:22,908 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:22,912 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:22,932 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testSnapshotsFromTables1 moved into observance of table space quota. 2023-07-12 19:40:22,936 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testSnapshotsFromTables0 moved into observance of table space quota. 2023-07-12 19:40:22,937 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x6f444882 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:22,941 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58094b9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:22,943 DEBUG [hconnection-0x502a9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:22,944 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34972, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:23,070 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:23,072 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:23,326 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. completed. 2023-07-12 19:40:23,326 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:40:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromTables0snapshot' locally completed 2023-07-12 19:40:23,327 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromTables0snapshot' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromTables0snapshot' has notified controller of completion 2023-07-12 19:40:23,330 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromTables0snapshot' completed. 2023-07-12 19:40:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromTables0snapshot' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:23,334 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromTables0snapshot' execution completed 2023-07-12 19:40:23,334 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:23,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromTables0snapshot', counting down latch. Waiting for 0 more 2023-07-12 19:40:23,335 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:23,335 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,335 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromTables0snapshotincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:23,340 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,340 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,340 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:23,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:23,341 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,342 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,350 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,350 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,350 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,350 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:23,350 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,350 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,350 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,350 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:23,350 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:23,350 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:23,351 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,351 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:23,353 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,380 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables0snapshot to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromTables0snapshot 2023-07-12 19:40:23,573 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:23,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:23,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:23,575 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:23,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x40568a6c to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:23,913 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@832680d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:23,915 DEBUG [hconnection-0x1ee08e53-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:23,917 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34974, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:24,209 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromTables0snapshot of table testSnapshotsFromTables0 completed 2023-07-12 19:40:24,209 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables0snapshot 2023-07-12 19:40:24,210 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables0snapshot 2023-07-12 19:40:24,211 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromTables0snapshot C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables0 at 1689190822458Consolidate snapshot: testSnapshotsFromTables0snapshot at 1689190823351 (+893 ms)Loading Region manifests for testSnapshotsFromTables0snapshot at 1689190823351Writing data manifest for testSnapshotsFromTables0snapshot at 1689190823361 (+10 ms)Verifying snapshot: testSnapshotsFromTables0snapshot at 1689190823371 (+10 ms)Snapshot testSnapshotsFromTables0snapshot of table testSnapshotsFromTables0 completed at 1689190824209 (+838 ms) 2023-07-12 19:40:24,212 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=22, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=SHARED 2023-07-12 19:40:24,214 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables0, type=SHARED in 1.7530 sec 2023-07-12 19:40:24,575 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:24,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:24,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromTables0snapshot table=testSnapshotsFromTables0 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:24,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:24,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190824578 (current time:1689190824578). 2023-07-12 19:40:24,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:24,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x2442864b to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:24,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4377f10c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:24,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:24,586 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2442864b to 127.0.0.1:57973 2023-07-12 19:40:24,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:24,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=23, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE 2023-07-12 19:40:24,591 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=23, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE 2023-07-12 19:40:24,592 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=23, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE; timeout=600000, timestamp=1689191424592 2023-07-12 19:40:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:24,592 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromTables1snapshot C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables1 2023-07-12 19:40:24,593 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:24,593 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:24,594 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=23, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE 2023-07-12 19:40:24,594 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED 2023-07-12 19:40:24,595 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE in 4 msec 2023-07-12 19:40:24,596 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=24, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED 2023-07-12 19:40:24,597 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=24, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED; timeout=600000, timestamp=1689191424596 2023-07-12 19:40:24,693 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:24,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:24,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:24,695 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:24,895 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:24,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:24,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:24,897 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:24,916 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x70554738 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:24,919 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSnapshotsFromTables1' 2023-07-12 19:40:24,920 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSnapshotsFromTables2' 2023-07-12 19:40:24,921 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b7e1ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:24,923 DEBUG [hconnection-0x5302beb2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:24,925 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:25,009 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,009 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromTables1snapshot' 2023-07-12 19:40:25,009 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:25,009 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromTables1snapshot' starting 'acquire' 2023-07-12 19:40:25,009 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromTables1snapshot', kicking off acquire phase on members. 2023-07-12 19:40:25,010 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,010 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,012 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,012 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:25,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:25,013 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,013 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 77 2023-07-12 19:40:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,014 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromTables1snapshot from table testSnapshotsFromTables1 type SKIPFLUSH 2023-07-12 19:40:25,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromTables1snapshot' with timeout 300000ms 2023-07-12 19:40:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromTables1snapshot' starting 'acquire' stage 2023-07-12 19:40:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromTables1snapshot' locally acquired 2023-07-12 19:40:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromTables1snapshot) in zk 2023-07-12 19:40:25,016 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,016 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromTables1snapshot' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:25,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromTables1snapshot' on coordinator 2023-07-12 19:40:25,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b84703[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:25,019 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromTables1snapshot' starting 'in-barrier' execution. 2023-07-12 19:40:25,019 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,020 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,020 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromTables1snapshot' received 'reached' from coordinator. 2023-07-12 19:40:25,020 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,020 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:25,020 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:25,020 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:25,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:25,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:25,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.' region-info for snapshot=testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:25,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:25,197 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:25,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:25,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:25,199 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:25,430 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. completed. 2023-07-12 19:40:25,430 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:25,430 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:25,430 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:25,430 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,430 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromTables1snapshot' locally completed 2023-07-12 19:40:25,430 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromTables1snapshot' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:25,433 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromTables1snapshot' has notified controller of completion 2023-07-12 19:40:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:25,433 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromTables1snapshot' completed. 2023-07-12 19:40:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromTables1snapshot' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:25,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromTables1snapshot', counting down latch. Waiting for 0 more 2023-07-12 19:40:25,437 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromTables1snapshot' execution completed 2023-07-12 19:40:25,437 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:25,437 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:25,437 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,437 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromTables1snapshotincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:25,439 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,439 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,439 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:25,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:25,440 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,442 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,449 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,449 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:25,449 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,449 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:25,449 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:25,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:25,449 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,449 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,450 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,450 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:25,450 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,450 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,450 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:25,451 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,466 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 19:40:25,699 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:25,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:25,701 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:25,871 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables1snapshot to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,910 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2b19cf23 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:25,914 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b968f91, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:25,919 DEBUG [hconnection-0x2348b7a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:25,921 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:25,962 DEBUG [master/jenkins-hbase4:0.Chore.1] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromTables1snapshot 2023-07-12 19:40:25,963 ERROR [master/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(166): Caught error java.lang.NullPointerException at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.getStoreFilesFromSnapshot(FileArchiverNotifierImpl.java:488) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeSnapshotSizes(FileArchiverNotifierImpl.java:404) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeAndStoreSnapshotSizes(FileArchiverNotifierImpl.java:334) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.computeSnapshotSizes(SnapshotQuotaObserverChore.java:225) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:118) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:26,696 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromTables1snapshot of table testSnapshotsFromTables1 completed 2023-07-12 19:40:26,696 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables1snapshot 2023-07-12 19:40:26,697 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables1snapshot 2023-07-12 19:40:26,697 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromTables1snapshot C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables1 at 1689190824592Consolidate snapshot: testSnapshotsFromTables1snapshot at 1689190825450 (+858 ms)Loading Region manifests for testSnapshotsFromTables1snapshot at 1689190825450Writing data manifest for testSnapshotsFromTables1snapshot at 1689190825455 (+5 ms)Verifying snapshot: testSnapshotsFromTables1snapshot at 1689190825864 (+409 ms)Snapshot testSnapshotsFromTables1snapshot of table testSnapshotsFromTables1 completed at 1689190826696 (+832 ms) 2023-07-12 19:40:26,698 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=24, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED 2023-07-12 19:40:26,700 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED in 2.1050 sec 2023-07-12 19:40:26,701 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:26,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:26,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromTables1snapshot table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:26,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:26,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190826704 (current time:1689190826704). 2023-07-12 19:40:26,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:26,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x2b9d9379 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:26,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@675e5579, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:26,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:26,711 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:26,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b9d9379 to 127.0.0.1:57973 2023-07-12 19:40:26,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:26,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:26,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:26,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=25, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE 2023-07-12 19:40:26,715 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=25, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE 2023-07-12 19:40:26,716 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=25, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE; timeout=600000, timestamp=1689191426716 2023-07-12 19:40:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:26,716 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromTables2snapshot C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables2 2023-07-12 19:40:26,717 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:26,717 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:26,718 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=26, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED 2023-07-12 19:40:26,718 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=25, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE 2023-07-12 19:40:26,719 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE in 4 msec 2023-07-12 19:40:26,719 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=26, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED 2023-07-12 19:40:26,720 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=26, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED; timeout=600000, timestamp=1689191426720 2023-07-12 19:40:26,817 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:26,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:26,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:26,819 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:26,917 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x6770eaac to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:26,923 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3055d02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:26,924 DEBUG [hconnection-0x2afa6072-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:26,926 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35022, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:27,019 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:27,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:27,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:27,021 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:27,132 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,132 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromTables2snapshot' 2023-07-12 19:40:27,132 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:27,132 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromTables2snapshot' starting 'acquire' 2023-07-12 19:40:27,132 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromTables2snapshot', kicking off acquire phase on members. 2023-07-12 19:40:27,132 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,132 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,134 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,134 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:27,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:27,134 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,134 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 77 2023-07-12 19:40:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,135 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromTables2snapshot from table testSnapshotsFromTables2 type SKIPFLUSH 2023-07-12 19:40:27,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromTables2snapshot' with timeout 300000ms 2023-07-12 19:40:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromTables2snapshot' starting 'acquire' stage 2023-07-12 19:40:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromTables2snapshot' locally acquired 2023-07-12 19:40:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromTables2snapshot) in zk 2023-07-12 19:40:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,138 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromTables2snapshot' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:27,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromTables2snapshot' on coordinator 2023-07-12 19:40:27,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@284c8c53[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:27,141 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromTables2snapshot' starting 'in-barrier' execution. 2023-07-12 19:40:27,141 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,142 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromTables2snapshot' received 'reached' from coordinator. 2023-07-12 19:40:27,143 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,143 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:27,143 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:27,143 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:27,143 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:27,143 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.' region-info for snapshot=testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,143 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:27,143 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:27,321 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:27,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:27,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:27,323 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:27,552 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. completed. 2023-07-12 19:40:27,552 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromTables2snapshot' locally completed 2023-07-12 19:40:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromTables2snapshot' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromTables2snapshot' has notified controller of completion 2023-07-12 19:40:27,557 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromTables2snapshot' completed. 2023-07-12 19:40:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromTables2snapshot' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:27,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromTables2snapshot', counting down latch. Waiting for 0 more 2023-07-12 19:40:27,560 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromTables2snapshot' execution completed 2023-07-12 19:40:27,560 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:27,560 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:27,560 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,560 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromTables2snapshotincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:27,562 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,562 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:27,562 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:27,563 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,565 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,568 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,568 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:27,568 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,568 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:27,568 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:27,568 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,568 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,569 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:27,569 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:27,569 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,569 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,569 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:27,570 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,587 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables2snapshot to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromTables2snapshot 2023-07-12 19:40:27,823 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:27,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:27,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:27,825 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:27,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4be392cb to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:27,912 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6083faf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:27,914 DEBUG [hconnection-0x7bccccb2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:27,916 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:28,414 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromTables2snapshot of table testSnapshotsFromTables2 completed 2023-07-12 19:40:28,414 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables2snapshot 2023-07-12 19:40:28,414 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables2snapshot 2023-07-12 19:40:28,414 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromTables2snapshot C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables2 at 1689190826716Consolidate snapshot: testSnapshotsFromTables2snapshot at 1689190827569 (+853 ms)Loading Region manifests for testSnapshotsFromTables2snapshot at 1689190827569Writing data manifest for testSnapshotsFromTables2snapshot at 1689190827573 (+4 ms)Verifying snapshot: testSnapshotsFromTables2snapshot at 1689190827581 (+8 ms)Snapshot testSnapshotsFromTables2snapshot of table testSnapshotsFromTables2 completed at 1689190828414 (+833 ms) 2023-07-12 19:40:28,415 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=26, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED 2023-07-12 19:40:28,418 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED in 1.6980 sec 2023-07-12 19:40:28,825 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:28,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:28,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromTables2snapshot table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:28,828 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x21689cdf to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:28,833 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54db4e07, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:28,835 DEBUG [hconnection-0x4cdaee04-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:28,839 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:28,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:28,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190828855 (current time:1689190828855). 2023-07-12 19:40:28,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:28,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x6b89b742 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:28,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2200e95f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:28,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:28,863 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:28,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b89b742 to 127.0.0.1:57973 2023-07-12 19:40:28,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:28,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:28,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:28,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE 2023-07-12 19:40:28,868 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=27, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE 2023-07-12 19:40:28,869 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=27, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE; timeout=600000, timestamp=1689191428869 2023-07-12 19:40:28,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:28,869 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromTables1snapshot1 C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables1 2023-07-12 19:40:28,869 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:28,869 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:28,870 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=27, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE 2023-07-12 19:40:28,870 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED 2023-07-12 19:40:28,872 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=EXCLUSIVE in 4 msec 2023-07-12 19:40:28,872 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=28, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED 2023-07-12 19:40:28,873 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=28, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED; timeout=600000, timestamp=1689191428873 2023-07-12 19:40:28,970 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:28,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:28,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:28,971 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:28,992 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5ab12cb3 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:28,999 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56fe4d35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:29,001 DEBUG [hconnection-0x3473f29d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:29,002 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:29,171 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:29,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:29,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:29,173 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:29,283 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,283 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromTables1snapshot1' 2023-07-12 19:40:29,283 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:29,283 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromTables1snapshot1' starting 'acquire' 2023-07-12 19:40:29,283 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromTables1snapshot1', kicking off acquire phase on members. 2023-07-12 19:40:29,284 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,284 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,285 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:29,285 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:29,285 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,286 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 78 2023-07-12 19:40:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,286 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromTables1snapshot1 from table testSnapshotsFromTables1 type SKIPFLUSH 2023-07-12 19:40:29,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,287 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromTables1snapshot1' with timeout 300000ms 2023-07-12 19:40:29,287 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:29,287 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromTables1snapshot1' starting 'acquire' stage 2023-07-12 19:40:29,287 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromTables1snapshot1' locally acquired 2023-07-12 19:40:29,287 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromTables1snapshot1) in zk 2023-07-12 19:40:29,289 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,289 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:29,290 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,290 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromTables1snapshot1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:29,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromTables1snapshot1' on coordinator 2023-07-12 19:40:29,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5081b81f[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:29,291 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromTables1snapshot1' starting 'in-barrier' execution. 2023-07-12 19:40:29,291 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,294 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,294 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromTables1snapshot1' received 'reached' from coordinator. 2023-07-12 19:40:29,294 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,294 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:29,294 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:29,294 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:29,294 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:29,294 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:29,294 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55.' region-info for snapshot=testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,294 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:29,294 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:29,473 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:29,475 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:29,703 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. completed. 2023-07-12 19:40:29,703 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:40:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromTables1snapshot1' locally completed 2023-07-12 19:40:29,704 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromTables1snapshot1' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:29,706 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromTables1snapshot1' has notified controller of completion 2023-07-12 19:40:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:29,706 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromTables1snapshot1' completed. 2023-07-12 19:40:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromTables1snapshot1' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:29,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromTables1snapshot1', counting down latch. Waiting for 0 more 2023-07-12 19:40:29,709 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromTables1snapshot1' execution completed 2023-07-12 19:40:29,710 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:29,710 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:29,710 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,710 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromTables1snapshot1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:29,712 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,712 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:29,712 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:29,712 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:29,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,714 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,717 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,717 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,718 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,718 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:29,718 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:29,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:29,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 because node does not exist (not an error) 2023-07-12 19:40:29,718 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,718 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,718 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:29,718 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,718 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:29,718 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:29,719 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:29,719 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:29,934 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x649ff042 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:29,948 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@181440c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:29,949 DEBUG [hconnection-0x57079490-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:29,952 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:29,975 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:29,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:29,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:29,977 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:30,139 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables1snapshot1 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:30,966 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromTables1snapshot1 of table testSnapshotsFromTables1 completed 2023-07-12 19:40:30,966 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:30,966 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables1snapshot1 2023-07-12 19:40:30,966 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromTables1snapshot1 C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables1 at 1689190828869Consolidate snapshot: testSnapshotsFromTables1snapshot1 at 1689190829718 (+849 ms)Loading Region manifests for testSnapshotsFromTables1snapshot1 at 1689190829718Writing data manifest for testSnapshotsFromTables1snapshot1 at 1689190829722 (+4 ms)Verifying snapshot: testSnapshotsFromTables1snapshot1 at 1689190830131 (+409 ms)Snapshot testSnapshotsFromTables1snapshot1 of table testSnapshotsFromTables1 completed at 1689190830966 (+835 ms) 2023-07-12 19:40:30,967 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=28, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED 2023-07-12 19:40:30,970 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables1, type=SHARED in 2.0980 sec 2023-07-12 19:40:30,977 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:30,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:30,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromTables1snapshot1 table=testSnapshotsFromTables1 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:30,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:30,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190830980 (current time:1689190830980). 2023-07-12 19:40:30,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:30,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x04ec7b16 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:30,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22f9ef6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:30,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:30,988 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50518, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04ec7b16 to 127.0.0.1:57973 2023-07-12 19:40:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:30,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:30,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:30,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=29, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE 2023-07-12 19:40:30,993 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=29, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE 2023-07-12 19:40:30,994 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=29, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE; timeout=600000, timestamp=1689191430994 2023-07-12 19:40:30,994 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromTables2snapshot1 C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables2 2023-07-12 19:40:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:30,995 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:30,995 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:30,996 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=30, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED 2023-07-12 19:40:30,996 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=29, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE 2023-07-12 19:40:30,998 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=EXCLUSIVE in 4 msec 2023-07-12 19:40:30,998 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=30, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED 2023-07-12 19:40:30,999 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7f5a0c0d to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:30,999 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=30, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED; timeout=600000, timestamp=1689191430999 2023-07-12 19:40:31,006 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a14747, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:31,010 DEBUG [hconnection-0x339a6483-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:31,011 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50524, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:31,095 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:31,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:31,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:31,097 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:31,297 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:31,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:31,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:31,299 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:31,414 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,414 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromTables2snapshot1' 2023-07-12 19:40:31,414 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:31,414 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromTables2snapshot1' starting 'acquire' 2023-07-12 19:40:31,414 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromTables2snapshot1', kicking off acquire phase on members. 2023-07-12 19:40:31,415 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,415 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,420 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:31,420 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:31,421 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,421 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:31,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 78 2023-07-12 19:40:31,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,422 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromTables2snapshot1 from table testSnapshotsFromTables2 type SKIPFLUSH 2023-07-12 19:40:31,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,422 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromTables2snapshot1' with timeout 300000ms 2023-07-12 19:40:31,422 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:31,422 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromTables2snapshot1' starting 'acquire' stage 2023-07-12 19:40:31,422 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromTables2snapshot1' locally acquired 2023-07-12 19:40:31,423 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromTables2snapshot1) in zk 2023-07-12 19:40:31,425 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,425 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:31,425 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,425 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromTables2snapshot1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:31,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromTables2snapshot1' on coordinator 2023-07-12 19:40:31,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33cde2eb[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:31,427 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromTables2snapshot1' starting 'in-barrier' execution. 2023-07-12 19:40:31,427 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,428 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,428 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromTables2snapshot1' received 'reached' from coordinator. 2023-07-12 19:40:31,429 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:31,429 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:31,429 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:31,429 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,429 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:31,429 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:31,429 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac.' region-info for snapshot=testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,429 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:31,429 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:31,599 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:31,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:31,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:31,601 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:31,838 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. completed. 2023-07-12 19:40:31,838 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:40:31,838 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:31,838 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:31,838 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,838 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromTables2snapshot1' locally completed 2023-07-12 19:40:31,838 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromTables2snapshot1' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:31,840 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromTables2snapshot1' has notified controller of completion 2023-07-12 19:40:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:31,840 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:31,841 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromTables2snapshot1' completed. 2023-07-12 19:40:31,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:31,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:31,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:31,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromTables2snapshot1' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:31,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromTables2snapshot1', counting down latch. Waiting for 0 more 2023-07-12 19:40:31,844 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromTables2snapshot1' execution completed 2023-07-12 19:40:31,844 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:31,845 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:31,845 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,845 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromTables2snapshot1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:31,847 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,847 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,848 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:31,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,848 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:31,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:31,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,849 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:31,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,852 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,852 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,852 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:31,852 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:31,852 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:31,852 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,853 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,852 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:31,853 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:31,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 because node does not exist (not an error) 2023-07-12 19:40:31,853 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,853 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,853 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:31,853 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:31,919 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5459642a to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:31,924 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41b09d53, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:31,925 DEBUG [hconnection-0x4c1593b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:31,927 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50532, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:32,101 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:32,103 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:32,273 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables2snapshot1 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:32,294 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromTables2snapshot1 of table testSnapshotsFromTables2 completed 2023-07-12 19:40:32,295 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:32,295 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromTables2snapshot1 2023-07-12 19:40:32,295 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromTables2snapshot1 C_M_SNAPSHOT_TABLE on table testSnapshotsFromTables2 at 1689190830994Consolidate snapshot: testSnapshotsFromTables2snapshot1 at 1689190831853 (+859 ms)Loading Region manifests for testSnapshotsFromTables2snapshot1 at 1689190831853Writing data manifest for testSnapshotsFromTables2snapshot1 at 1689190831856 (+3 ms)Verifying snapshot: testSnapshotsFromTables2snapshot1 at 1689190832266 (+410 ms)Snapshot testSnapshotsFromTables2snapshot1 of table testSnapshotsFromTables2 completed at 1689190832294 (+28 ms) 2023-07-12 19:40:32,296 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=30, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED 2023-07-12 19:40:32,298 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromTables2, type=SHARED in 1.3010 sec 2023-07-12 19:40:32,912 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7211b91b to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:32,917 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37ebd314, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:32,919 DEBUG [hconnection-0x1bd2f82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:32,921 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:33,104 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:33,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:33,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromTables2snapshot1 table=testSnapshotsFromTables2 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:33,106 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x442f6434 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:33,111 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50c76eee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:33,113 DEBUG [hconnection-0x64bec5a3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:33,116 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50546, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:33,157 INFO [Listener at localhost/32953] hbase.ResourceChecker(175): after: quotas.TestSnapshotQuotaObserverChore#testSnapshotsFromTables Thread=283 (was 212) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x5459642a-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x3e827ec6-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x1a46a97a-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x21689cdf-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x1bd2f82-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x21689cdf sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7211b91b-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x6770eaac-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x36499595-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x2b19cf23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x2b19cf23-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x1bd2f82-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e827ec6-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x64bec5a3-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x1a46a97a-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x4c1593b1-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7211b91b-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x40568a6c-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x3e827ec6-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x442f6434-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x36499595 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x70554738-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x649ff042-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x70554738 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/cluster_cb8e40ba-4554-a1e8-3e84-9973d51e9321/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x59c16f00-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7f5a0c0d-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: (jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x64bec5a3-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x4be392cb-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7f5a0c0d-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7211b91b sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e827ec6-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x649ff042 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/cluster_cb8e40ba-4554-a1e8-3e84-9973d51e9321/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x1d23dcc6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x5ab12cb3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x59c16f00-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x4be392cb sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x21689cdf-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x4c1593b1-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x2b19cf23-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x6f444882-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x59c16f00-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x6f444882-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x1d23dcc6-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x36499595-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x40568a6c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x5459642a sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x70554738-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-361464297_17 at /127.0.0.1:58402 [Waiting for operation #58] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x339a6483-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-361464297_17 at /127.0.0.1:47788 [Waiting for operation #52] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x40568a6c sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x442f6434-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x442f6434 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x339a6483-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x1d23dcc6-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x3e827ec6-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x5ab12cb3-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x5ab12cb3-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7f5a0c0d sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x649ff042-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x6770eaac-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x6770eaac sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x5459642a-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x4be392cb-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e827ec6-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x1a46a97a sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x6f444882 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=592 (was 473) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=104 (was 124), ProcessCount=172 (was 172), AvailableMemoryMB=8023 (was 8210) 2023-07-12 19:40:33,171 INFO [Listener at localhost/32953] hbase.ResourceChecker(147): before: quotas.TestSnapshotQuotaObserverChore#testSnapshotSize Thread=283, OpenFileDescriptor=592, MaxFileDescriptor=60000, SystemLoadAverage=104, ProcessCount=172, AvailableMemoryMB=8022 2023-07-12 19:40:33,174 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x290bf6f1 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:33,178 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4938f23c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:33,180 DEBUG [hconnection-0x6e1e2d59-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:33,182 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:33,186 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(447): Deleting quota for table: testSnapshotsFromTables0 2023-07-12 19:40:33,191 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(447): Deleting quota for table: testSnapshotsFromTables1 2023-07-12 19:40:33,192 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x290bf6f1 to 127.0.0.1:57973 2023-07-12 19:40:33,192 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:33,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testSnapshotSize3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:33,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSnapshotSize3 2023-07-12 19:40:33,199 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=31, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSnapshotSize3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:33,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testSnapshotSize3" procId is: 31 2023-07-12 19:40:33,200 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=31, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSnapshotSize3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:33,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 19:40:33,205 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:33,205 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:33,205 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:33,205 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:33,205 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:33,205 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615 empty. 2023-07-12 19:40:33,205 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c empty. 2023-07-12 19:40:33,206 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07 empty. 2023-07-12 19:40:33,206 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae empty. 2023-07-12 19:40:33,206 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83 empty. 2023-07-12 19:40:33,206 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:33,206 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:33,206 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:33,207 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:33,207 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:33,207 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSnapshotSize3 regions 2023-07-12 19:40:33,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 19:40:33,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 19:40:33,617 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotSize3/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:33,618 INFO [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(7675): creating {ENCODED => 4f6eda388e14a12df47ea1363fceed07, NAME => 'testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07.', STARTKEY => '3', ENDKEY => '6'}, tableDescriptor='testSnapshotSize3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:33,618 INFO [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(7675): creating {ENCODED => b7f092ed0d4ff500eb86950c457f0615, NAME => 'testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615.', STARTKEY => '0', ENDKEY => '3'}, tableDescriptor='testSnapshotSize3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:33,618 INFO [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0c1b9295d50da7c1350cece1d5853d2c, NAME => 'testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c.', STARTKEY => '', ENDKEY => '0'}, tableDescriptor='testSnapshotSize3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:33,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 19:40:33,984 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x20750642 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:33,988 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c143efc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:33,990 DEBUG [hconnection-0x6e441045-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:33,992 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:34,042 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(866): Instantiated testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,042 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1604): Closing 4f6eda388e14a12df47ea1363fceed07, disabling compactions & flushes 2023-07-12 19:40:34,042 INFO [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1626): Closing region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:34,042 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:34,042 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. after waiting 0 ms 2023-07-12 19:40:34,042 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:34,042 INFO [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1838): Closed testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:34,042 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1558): Region close journal for 4f6eda388e14a12df47ea1363fceed07: 2023-07-12 19:40:34,043 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(866): Instantiated testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,043 INFO [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(7675): creating {ENCODED => 9869f769c7045fb07f1013a482eb9f83, NAME => 'testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83.', STARTKEY => '6', ENDKEY => '9'}, tableDescriptor='testSnapshotSize3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:34,043 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1604): Closing b7f092ed0d4ff500eb86950c457f0615, disabling compactions & flushes 2023-07-12 19:40:34,043 INFO [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1626): Closing region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:34,043 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:34,043 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. after waiting 0 ms 2023-07-12 19:40:34,043 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:34,043 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(866): Instantiated testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,043 INFO [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1838): Closed testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:34,044 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(1604): Closing 0c1b9295d50da7c1350cece1d5853d2c, disabling compactions & flushes 2023-07-12 19:40:34,044 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1558): Region close journal for b7f092ed0d4ff500eb86950c457f0615: 2023-07-12 19:40:34,044 INFO [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(1626): Closing region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:34,044 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:34,044 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. after waiting 0 ms 2023-07-12 19:40:34,044 INFO [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(7675): creating {ENCODED => 2eb66a3eec48e0be370ce75cea0caeae, NAME => 'testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae.', STARTKEY => '9', ENDKEY => ''}, tableDescriptor='testSnapshotSize3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:34,044 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:34,044 INFO [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(1838): Closed testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:34,045 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-0] regionserver.HRegion(1558): Region close journal for 0c1b9295d50da7c1350cece1d5853d2c: 2023-07-12 19:40:34,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(866): Instantiated testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(866): Instantiated testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1604): Closing 9869f769c7045fb07f1013a482eb9f83, disabling compactions & flushes 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1604): Closing 2eb66a3eec48e0be370ce75cea0caeae, disabling compactions & flushes 2023-07-12 19:40:34,451 INFO [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1626): Closing region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:34,451 INFO [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1626): Closing region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. after waiting 0 ms 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. after waiting 0 ms 2023-07-12 19:40:34,451 INFO [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1838): Closed testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:34,451 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-2] regionserver.HRegion(1558): Region close journal for 9869f769c7045fb07f1013a482eb9f83: 2023-07-12 19:40:34,452 INFO [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1838): Closed testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:34,452 DEBUG [RegionOpenAndInit-testSnapshotSize3-pool-1] regionserver.HRegion(1558): Region close journal for 2eb66a3eec48e0be370ce75cea0caeae: 2023-07-12 19:40:34,455 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=31, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSnapshotSize3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:34,456 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834456"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190834456"}]},"ts":"1689190834456"} 2023-07-12 19:40:34,456 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834456"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190834456"}]},"ts":"1689190834456"} 2023-07-12 19:40:34,456 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689190834456"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190834456"}]},"ts":"1689190834456"} 2023-07-12 19:40:34,456 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834456"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190834456"}]},"ts":"1689190834456"} 2023-07-12 19:40:34,456 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689190834456"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190834456"}]},"ts":"1689190834456"} 2023-07-12 19:40:34,459 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 5 regions to meta. 2023-07-12 19:40:34,460 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=31, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSnapshotSize3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:34,460 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotSize3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190834460"}]},"ts":"1689190834460"} 2023-07-12 19:40:34,462 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotSize3, state=ENABLING in hbase:meta 2023-07-12 19:40:34,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=0c1b9295d50da7c1350cece1d5853d2c, ASSIGN}, {pid=33, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=b7f092ed0d4ff500eb86950c457f0615, ASSIGN}, {pid=34, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=4f6eda388e14a12df47ea1363fceed07, ASSIGN}, {pid=35, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=9869f769c7045fb07f1013a482eb9f83, ASSIGN}, {pid=36, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=2eb66a3eec48e0be370ce75cea0caeae, ASSIGN}] 2023-07-12 19:40:34,468 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=34, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=4f6eda388e14a12df47ea1363fceed07, ASSIGN 2023-07-12 19:40:34,469 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=33, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=b7f092ed0d4ff500eb86950c457f0615, ASSIGN 2023-07-12 19:40:34,469 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=0c1b9295d50da7c1350cece1d5853d2c, ASSIGN 2023-07-12 19:40:34,469 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=35, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=9869f769c7045fb07f1013a482eb9f83, ASSIGN 2023-07-12 19:40:34,470 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=34, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotSize3, region=4f6eda388e14a12df47ea1363fceed07, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:34,470 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=33, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotSize3, region=b7f092ed0d4ff500eb86950c457f0615, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:34,470 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotSize3, region=0c1b9295d50da7c1350cece1d5853d2c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:34,470 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=35, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotSize3, region=9869f769c7045fb07f1013a482eb9f83, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:34,470 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotSize3, region=2eb66a3eec48e0be370ce75cea0caeae, ASSIGN 2023-07-12 19:40:34,471 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotSize3, region=2eb66a3eec48e0be370ce75cea0caeae, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:34,622 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=34 updating hbase:meta row=4f6eda388e14a12df47ea1363fceed07, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,622 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=35 updating hbase:meta row=9869f769c7045fb07f1013a482eb9f83, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,622 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=33 updating hbase:meta row=b7f092ed0d4ff500eb86950c457f0615, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,622 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=2eb66a3eec48e0be370ce75cea0caeae, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,622 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=0c1b9295d50da7c1350cece1d5853d2c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,622 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190834622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190834622"}]},"ts":"1689190834622"} 2023-07-12 19:40:34,622 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190834622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190834622"}]},"ts":"1689190834622"} 2023-07-12 19:40:34,622 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190834622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190834622"}]},"ts":"1689190834622"} 2023-07-12 19:40:34,622 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689190834622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190834622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190834622"}]},"ts":"1689190834622"} 2023-07-12 19:40:34,622 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689190834622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190834622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190834622"}]},"ts":"1689190834622"} 2023-07-12 19:40:34,625 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=33, state=RUNNABLE; OpenRegionProcedure b7f092ed0d4ff500eb86950c457f0615, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:34,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=38, ppid=35, state=RUNNABLE; OpenRegionProcedure 9869f769c7045fb07f1013a482eb9f83, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:34,627 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=34, state=RUNNABLE; OpenRegionProcedure 4f6eda388e14a12df47ea1363fceed07, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:34,628 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=32, state=RUNNABLE; OpenRegionProcedure 0c1b9295d50da7c1350cece1d5853d2c, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:34,629 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=36, state=RUNNABLE; OpenRegionProcedure 2eb66a3eec48e0be370ce75cea0caeae, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:34,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:34,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9869f769c7045fb07f1013a482eb9f83, NAME => 'testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83.', STARTKEY => '6', ENDKEY => '9'} 2023-07-12 19:40:34,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotSize3 9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:34,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:34,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:34,784 INFO [StoreOpener-9869f769c7045fb07f1013a482eb9f83-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:34,785 DEBUG [StoreOpener-9869f769c7045fb07f1013a482eb9f83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1 2023-07-12 19:40:34,785 DEBUG [StoreOpener-9869f769c7045fb07f1013a482eb9f83-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1 2023-07-12 19:40:34,786 INFO [StoreOpener-9869f769c7045fb07f1013a482eb9f83-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9869f769c7045fb07f1013a482eb9f83 columnFamilyName f1 2023-07-12 19:40:34,787 INFO [StoreOpener-9869f769c7045fb07f1013a482eb9f83-1] regionserver.HStore(310): Store=9869f769c7045fb07f1013a482eb9f83/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:34,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:34,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:34,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:40:34,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:34,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9869f769c7045fb07f1013a482eb9f83; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75394023, jitterRate=0.12345848977565765}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:34,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9869f769c7045fb07f1013a482eb9f83: 2023-07-12 19:40:34,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83., pid=38, masterSystemTime=1689190834777 2023-07-12 19:40:34,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:34,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:34,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:34,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4f6eda388e14a12df47ea1363fceed07, NAME => 'testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07.', STARTKEY => '3', ENDKEY => '6'} 2023-07-12 19:40:34,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotSize3 4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:34,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:34,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:34,798 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=35 updating hbase:meta row=9869f769c7045fb07f1013a482eb9f83, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,798 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834798"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190834798"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190834798"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190834798"}]},"ts":"1689190834798"} 2023-07-12 19:40:34,799 INFO [StoreOpener-4f6eda388e14a12df47ea1363fceed07-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:34,801 DEBUG [StoreOpener-4f6eda388e14a12df47ea1363fceed07-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1 2023-07-12 19:40:34,801 DEBUG [StoreOpener-4f6eda388e14a12df47ea1363fceed07-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1 2023-07-12 19:40:34,801 INFO [StoreOpener-4f6eda388e14a12df47ea1363fceed07-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4f6eda388e14a12df47ea1363fceed07 columnFamilyName f1 2023-07-12 19:40:34,802 INFO [StoreOpener-4f6eda388e14a12df47ea1363fceed07-1] regionserver.HStore(310): Store=4f6eda388e14a12df47ea1363fceed07/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:34,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:34,804 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=38, resume processing ppid=35 2023-07-12 19:40:34,804 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=35, state=SUCCESS; OpenRegionProcedure 9869f769c7045fb07f1013a482eb9f83, server=jenkins-hbase4.apache.org,44631,1689190811374 in 175 msec 2023-07-12 19:40:34,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:34,805 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotSize3, region=9869f769c7045fb07f1013a482eb9f83, ASSIGN in 338 msec 2023-07-12 19:40:34,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:40:34,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:34,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4f6eda388e14a12df47ea1363fceed07; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75358800, jitterRate=0.12293362617492676}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:34,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4f6eda388e14a12df47ea1363fceed07: 2023-07-12 19:40:34,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07., pid=39, masterSystemTime=1689190834777 2023-07-12 19:40:34,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:34,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:34,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b7f092ed0d4ff500eb86950c457f0615, NAME => 'testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615.', STARTKEY => '0', ENDKEY => '3'} 2023-07-12 19:40:34,813 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=34 updating hbase:meta row=4f6eda388e14a12df47ea1363fceed07, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotSize3 b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,813 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834813"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190834813"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190834813"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190834813"}]},"ts":"1689190834813"} 2023-07-12 19:40:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:34,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:34,815 INFO [StoreOpener-b7f092ed0d4ff500eb86950c457f0615-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:34,817 DEBUG [StoreOpener-b7f092ed0d4ff500eb86950c457f0615-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1 2023-07-12 19:40:34,817 DEBUG [StoreOpener-b7f092ed0d4ff500eb86950c457f0615-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1 2023-07-12 19:40:34,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=34 2023-07-12 19:40:34,818 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=34, state=SUCCESS; OpenRegionProcedure 4f6eda388e14a12df47ea1363fceed07, server=jenkins-hbase4.apache.org,44631,1689190811374 in 189 msec 2023-07-12 19:40:34,819 INFO [StoreOpener-b7f092ed0d4ff500eb86950c457f0615-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b7f092ed0d4ff500eb86950c457f0615 columnFamilyName f1 2023-07-12 19:40:34,820 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotSize3, region=4f6eda388e14a12df47ea1363fceed07, ASSIGN in 352 msec 2023-07-12 19:40:34,820 INFO [StoreOpener-b7f092ed0d4ff500eb86950c457f0615-1] regionserver.HStore(310): Store=b7f092ed0d4ff500eb86950c457f0615/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:34,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:34,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:34,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:40:34,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:34,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b7f092ed0d4ff500eb86950c457f0615; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61463852, jitterRate=-0.08411723375320435}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:34,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b7f092ed0d4ff500eb86950c457f0615: 2023-07-12 19:40:34,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615., pid=37, masterSystemTime=1689190834777 2023-07-12 19:40:34,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:34,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:34,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:34,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0c1b9295d50da7c1350cece1d5853d2c, NAME => 'testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c.', STARTKEY => '', ENDKEY => '0'} 2023-07-12 19:40:34,831 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=33 updating hbase:meta row=b7f092ed0d4ff500eb86950c457f0615, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotSize3 0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:34,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,831 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1689190834831"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190834831"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190834831"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190834831"}]},"ts":"1689190834831"} 2023-07-12 19:40:34,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:34,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:34,833 INFO [StoreOpener-0c1b9295d50da7c1350cece1d5853d2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:34,837 DEBUG [StoreOpener-0c1b9295d50da7c1350cece1d5853d2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c/f1 2023-07-12 19:40:34,837 DEBUG [StoreOpener-0c1b9295d50da7c1350cece1d5853d2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c/f1 2023-07-12 19:40:34,838 INFO [StoreOpener-0c1b9295d50da7c1350cece1d5853d2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0c1b9295d50da7c1350cece1d5853d2c columnFamilyName f1 2023-07-12 19:40:34,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=33 2023-07-12 19:40:34,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=33, state=SUCCESS; OpenRegionProcedure b7f092ed0d4ff500eb86950c457f0615, server=jenkins-hbase4.apache.org,44631,1689190811374 in 211 msec 2023-07-12 19:40:34,839 INFO [StoreOpener-0c1b9295d50da7c1350cece1d5853d2c-1] regionserver.HStore(310): Store=0c1b9295d50da7c1350cece1d5853d2c/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:34,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:34,840 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotSize3, region=b7f092ed0d4ff500eb86950c457f0615, ASSIGN in 373 msec 2023-07-12 19:40:34,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:34,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:40:34,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:34,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0c1b9295d50da7c1350cece1d5853d2c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59017915, jitterRate=-0.1205645352602005}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:34,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0c1b9295d50da7c1350cece1d5853d2c: 2023-07-12 19:40:34,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c., pid=40, masterSystemTime=1689190834777 2023-07-12 19:40:34,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:34,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:34,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:34,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2eb66a3eec48e0be370ce75cea0caeae, NAME => 'testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae.', STARTKEY => '9', ENDKEY => ''} 2023-07-12 19:40:34,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotSize3 2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:34,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:34,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:34,850 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=0c1b9295d50da7c1350cece1d5853d2c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:34,850 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689190834850"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190834850"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190834850"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190834850"}]},"ts":"1689190834850"} 2023-07-12 19:40:34,851 INFO [StoreOpener-2eb66a3eec48e0be370ce75cea0caeae-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:34,853 DEBUG [StoreOpener-2eb66a3eec48e0be370ce75cea0caeae-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1 2023-07-12 19:40:34,853 DEBUG [StoreOpener-2eb66a3eec48e0be370ce75cea0caeae-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1 2023-07-12 19:40:34,854 INFO [StoreOpener-2eb66a3eec48e0be370ce75cea0caeae-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2eb66a3eec48e0be370ce75cea0caeae columnFamilyName f1 2023-07-12 19:40:34,854 INFO [StoreOpener-2eb66a3eec48e0be370ce75cea0caeae-1] regionserver.HStore(310): Store=2eb66a3eec48e0be370ce75cea0caeae/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:34,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=32 2023-07-12 19:40:34,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=32, state=SUCCESS; OpenRegionProcedure 0c1b9295d50da7c1350cece1d5853d2c, server=jenkins-hbase4.apache.org,44631,1689190811374 in 224 msec 2023-07-12 19:40:34,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:34,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:34,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotSize3, region=0c1b9295d50da7c1350cece1d5853d2c, ASSIGN in 389 msec 2023-07-12 19:40:34,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:40:34,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:34,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2eb66a3eec48e0be370ce75cea0caeae; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68441159, jitterRate=0.019852742552757263}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:34,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2eb66a3eec48e0be370ce75cea0caeae: 2023-07-12 19:40:34,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae., pid=41, masterSystemTime=1689190834777 2023-07-12 19:40:34,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:34,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:34,865 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=2eb66a3eec48e0be370ce75cea0caeae, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:34,866 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1689190834865"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190834865"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190834865"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190834865"}]},"ts":"1689190834865"} 2023-07-12 19:40:34,870 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=36 2023-07-12 19:40:34,870 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; OpenRegionProcedure 2eb66a3eec48e0be370ce75cea0caeae, server=jenkins-hbase4.apache.org,44631,1689190811374 in 239 msec 2023-07-12 19:40:34,873 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=31 2023-07-12 19:40:34,873 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotSize3, region=2eb66a3eec48e0be370ce75cea0caeae, ASSIGN in 404 msec 2023-07-12 19:40:34,874 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=31, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSnapshotSize3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:34,874 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotSize3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190834874"}]},"ts":"1689190834874"} 2023-07-12 19:40:34,876 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotSize3, state=ENABLED in hbase:meta 2023-07-12 19:40:34,879 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=31, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSnapshotSize3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:34,881 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; CreateTableProcedure table=testSnapshotSize3 in 1.6830 sec 2023-07-12 19:40:34,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1abe42a0 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:34,913 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20e6508b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:34,917 DEBUG [hconnection-0x5554a384-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:34,920 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:35,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=31 2023-07-12 19:40:35,309 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testSnapshotSize3, procId: 31 completed 2023-07-12 19:40:35,329 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 19:40:35,330 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=MasterService 2023-07-12 19:40:35,446 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 19:40:35,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(936): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-07-12 19:40:35,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotSize3 2023-07-12 19:40:35,461 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotSize3' 2023-07-12 19:40:35,461 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:35,462 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotSize3' starting 'acquire' 2023-07-12 19:40:35,462 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotSize3', kicking off acquire phase on members. 2023-07-12 19:40:35,462 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:35,462 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:35,469 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,469 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:35,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:35,469 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,469 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:35,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 19:40:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:35,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSnapshotSize3 2023-07-12 19:40:35,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotSize3 2023-07-12 19:40:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotSize3' with timeout 60000ms 2023-07-12 19:40:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotSize3' starting 'acquire' stage 2023-07-12 19:40:35,473 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:35,473 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 5 regions 2023-07-12 19:40:35,473 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. started... 2023-07-12 19:40:35,473 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:35,473 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 19:40:35,473 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. started... 2023-07-12 19:40:35,473 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:35,474 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. started... 2023-07-12 19:40:35,475 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9869f769c7045fb07f1013a482eb9f83 1/1 column families, dataSize=84.54 KB heapSize=107.83 KB 2023-07-12 19:40:35,475 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing b7f092ed0d4ff500eb86950c457f0615 1/1 column families, dataSize=85.37 KB heapSize=108.88 KB 2023-07-12 19:40:35,476 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 0c1b9295d50da7c1350cece1d5853d2c: 2023-07-12 19:40:35,476 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:35,477 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:35,477 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/5 local region flush tasks. 2023-07-12 19:40:35,477 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. started... 2023-07-12 19:40:35,477 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2745): Flushing 4f6eda388e14a12df47ea1363fceed07 1/1 column families, dataSize=84.82 KB heapSize=108.18 KB 2023-07-12 19:40:35,563 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=85.37 KB at sequenceid=25 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/.tmp/f1/92ccedf11cef4cf884248c39062924a2 2023-07-12 19:40:35,618 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/.tmp/f1/92ccedf11cef4cf884248c39062924a2 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/92ccedf11cef4cf884248c39062924a2 2023-07-12 19:40:35,625 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/92ccedf11cef4cf884248c39062924a2, entries=309, sequenceid=25, filesize=91.0 K 2023-07-12 19:40:35,628 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~85.37 KB/87423, heapSize ~108.87 KB/111480, currentSize=0 B/0 for b7f092ed0d4ff500eb86950c457f0615 in 152ms, sequenceid=25, compaction requested=false 2023-07-12 19:40:35,628 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSnapshotSize3' 2023-07-12 19:40:35,628 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for b7f092ed0d4ff500eb86950c457f0615: 2023-07-12 19:40:35,628 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:35,628 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:35,628 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 2/5 local region flush tasks. 2023-07-12 19:40:35,629 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. started... 2023-07-12 19:40:35,629 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing 2eb66a3eec48e0be370ce75cea0caeae 1/1 column families, dataSize=28.18 KB heapSize=36.11 KB 2023-07-12 19:40:35,639 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=28.18 KB at sequenceid=24 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/.tmp/f1/c68e1caa97f044da8a80480dc854370a 2023-07-12 19:40:35,645 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/.tmp/f1/c68e1caa97f044da8a80480dc854370a as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/c68e1caa97f044da8a80480dc854370a 2023-07-12 19:40:35,651 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/c68e1caa97f044da8a80480dc854370a, entries=102, sequenceid=24, filesize=33.1 K 2023-07-12 19:40:35,652 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~28.18 KB/28857, heapSize ~36.09 KB/36960, currentSize=0 B/0 for 2eb66a3eec48e0be370ce75cea0caeae in 22ms, sequenceid=24, compaction requested=false 2023-07-12 19:40:35,652 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for 2eb66a3eec48e0be370ce75cea0caeae: 2023-07-12 19:40:35,652 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:35,652 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 3/5 local region flush tasks. 2023-07-12 19:40:35,935 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7c1d91f6 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:35,944 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f22e641, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:35,946 DEBUG [hconnection-0xd70add7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:35,948 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:35,964 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=84.54 KB at sequenceid=24 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/.tmp/f1/d0115a6cb0a849cea1235a2f7a35f53e 2023-07-12 19:40:35,969 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=84.82 KB at sequenceid=25 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/.tmp/f1/f1ee2f9dba5847a2ab1da179375d8ba1 2023-07-12 19:40:35,971 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/.tmp/f1/d0115a6cb0a849cea1235a2f7a35f53e as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/d0115a6cb0a849cea1235a2f7a35f53e 2023-07-12 19:40:35,976 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/.tmp/f1/f1ee2f9dba5847a2ab1da179375d8ba1 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/f1ee2f9dba5847a2ab1da179375d8ba1 2023-07-12 19:40:35,977 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/d0115a6cb0a849cea1235a2f7a35f53e, entries=306, sequenceid=24, filesize=90.1 K 2023-07-12 19:40:35,978 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~84.54 KB/86571, heapSize ~107.81 KB/110400, currentSize=0 B/0 for 9869f769c7045fb07f1013a482eb9f83 in 503ms, sequenceid=24, compaction requested=false 2023-07-12 19:40:35,979 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9869f769c7045fb07f1013a482eb9f83: 2023-07-12 19:40:35,979 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:35,979 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 4/5 local region flush tasks. 2023-07-12 19:40:35,982 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/f1ee2f9dba5847a2ab1da179375d8ba1, entries=307, sequenceid=25, filesize=90.4 K 2023-07-12 19:40:35,983 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2948): Finished flush of dataSize ~84.82 KB/86855, heapSize ~108.16 KB/110760, currentSize=0 B/0 for 4f6eda388e14a12df47ea1363fceed07 in 506ms, sequenceid=25, compaction requested=false 2023-07-12 19:40:35,983 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 4f6eda388e14a12df47ea1363fceed07: 2023-07-12 19:40:35,983 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:35,983 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 5/5 local region flush tasks. 2023-07-12 19:40:35,983 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 5 local region flush tasks. 2023-07-12 19:40:35,984 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,984 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotSize3' locally acquired 2023-07-12 19:40:35,984 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotSize3) in zk 2023-07-12 19:40:35,985 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:35,985 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:35,986 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:35,986 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotSize3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:35,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotSize3' on coordinator 2023-07-12 19:40:35,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79873bce[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:35,987 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotSize3' starting 'in-barrier' execution. 2023-07-12 19:40:35,987 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:35,989 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:35,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:35,989 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotSize3' received 'reached' from coordinator. 2023-07-12 19:40:35,990 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,990 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:35,990 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotSize3' locally completed 2023-07-12 19:40:35,990 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotSize3' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:35,991 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotSize3' has notified controller of completion 2023-07-12 19:40:35,991 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,991 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,991 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotSize3' completed. 2023-07-12 19:40:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:35,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotSize3' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:35,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotSize3', counting down latch. Waiting for 0 more 2023-07-12 19:40:35,995 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotSize3' execution completed 2023-07-12 19:40:35,995 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:35,995 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:35,995 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotSize3 2023-07-12 19:40:35,995 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotSize3including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 19:40:35,997 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:35,997 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:35,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:35,997 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:35,998 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:35,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:35,999 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:36,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,001 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,002 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:36,002 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:36,002 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,002 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,002 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:36,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSnapshotSize3' 2023-07-12 19:40:36,002 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:36,002 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,002 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:36,002 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:36,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 19:40:36,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSnapshotSize3 because node does not exist (not an error) 2023-07-12 19:40:36,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:36,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:36,004 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSnapshotSize3'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:36,004 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 19:40:36,104 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 19:40:36,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 19:40:36,111 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 19:40:36,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(936): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-07-12 19:40:36,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(143): Procedure testSnapshotSize3 was in running list but was completed. Accepting new attempt. 2023-07-12 19:40:36,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotSize3 2023-07-12 19:40:36,117 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotSize3' 2023-07-12 19:40:36,118 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:36,118 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotSize3' starting 'acquire' 2023-07-12 19:40:36,118 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotSize3', kicking off acquire phase on members. 2023-07-12 19:40:36,119 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,119 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:36,120 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,120 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:36,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:36,120 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,120 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:36,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:36,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 19:40:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:36,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSnapshotSize3 2023-07-12 19:40:36,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSnapshotSize3 is still present, removing 2023-07-12 19:40:36,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotSize3 2023-07-12 19:40:36,121 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotSize3' with timeout 60000ms 2023-07-12 19:40:36,121 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:36,122 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotSize3' starting 'acquire' stage 2023-07-12 19:40:36,122 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:36,122 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. started... 2023-07-12 19:40:36,122 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 5 regions 2023-07-12 19:40:36,122 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:36,122 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9869f769c7045fb07f1013a482eb9f83: 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:36,122 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 19:40:36,122 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:36,123 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/5 local region flush tasks. 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. started... 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. started... 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. started... 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4f6eda388e14a12df47ea1363fceed07: 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for b7f092ed0d4ff500eb86950c457f0615: 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:36,123 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 2/5 local region flush tasks. 2023-07-12 19:40:36,123 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 3/5 local region flush tasks. 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. started... 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 0c1b9295d50da7c1350cece1d5853d2c: 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2eb66a3eec48e0be370ce75cea0caeae: 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:36,123 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:36,123 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 4/5 local region flush tasks. 2023-07-12 19:40:36,124 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 5/5 local region flush tasks. 2023-07-12 19:40:36,124 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 5 local region flush tasks. 2023-07-12 19:40:36,124 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,124 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotSize3' locally acquired 2023-07-12 19:40:36,124 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotSize3) in zk 2023-07-12 19:40:36,125 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,125 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:36,126 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:36,126 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotSize3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:36,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotSize3' on coordinator 2023-07-12 19:40:36,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35651eb9[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:36,127 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotSize3' starting 'in-barrier' execution. 2023-07-12 19:40:36,127 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,129 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,130 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,130 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotSize3' received 'reached' from coordinator. 2023-07-12 19:40:36,130 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:36,130 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotSize3' locally completed 2023-07-12 19:40:36,130 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotSize3' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:36,132 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,132 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotSize3' has notified controller of completion 2023-07-12 19:40:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:36,132 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:36,132 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotSize3' completed. 2023-07-12 19:40:36,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotSize3' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:36,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotSize3', counting down latch. Waiting for 0 more 2023-07-12 19:40:36,136 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotSize3' execution completed 2023-07-12 19:40:36,136 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:36,136 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:36,136 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotSize3 2023-07-12 19:40:36,136 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotSize3including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 19:40:36,137 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,138 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,138 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:36,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,138 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:36,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:36,140 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:36,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:36,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,143 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,143 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:36,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSnapshotSize3' 2023-07-12 19:40:36,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 19:40:36,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSnapshotSize3 because node does not exist (not an error) 2023-07-12 19:40:36,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:36,143 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:36,143 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:36,144 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSnapshotSize3'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:36,143 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:36,144 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 19:40:36,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:36,144 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:36,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:36,144 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:36,144 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,145 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:36,145 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:36,244 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 19:40:36,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 19:40:36,248 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:40:36,259 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: null 2023-07-12 19:40:36,360 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: null 2023-07-12 19:40:36,462 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: null 2023-07-12 19:40:36,564 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: null 2023-07-12 19:40:36,666 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: null 2023-07-12 19:40:36,768 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: null 2023-07-12 19:40:36,870 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: null 2023-07-12 19:40:36,915 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testSnapshotSize3 moved into observance of table space quota. 2023-07-12 19:40:36,915 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7792e9a5 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:36,921 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d138f5f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:36,924 DEBUG [hconnection-0x13ebc377-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:36,925 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:36,972 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=304.64 KB/1 GB] 2023-07-12 19:40:36,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:36,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190836973 (current time:1689190836973). 2023-07-12 19:40:36,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:36,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x16b2c1bb to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:36,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@516c597, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:36,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:36,981 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:36,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16b2c1bb to 127.0.0.1:57973 2023-07-12 19:40:36,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:36,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:36,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:36,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=42, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=EXCLUSIVE 2023-07-12 19:40:36,986 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=42, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=EXCLUSIVE 2023-07-12 19:40:36,987 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=42, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=EXCLUSIVE; timeout=600000, timestamp=1689191436987 2023-07-12 19:40:36,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:36,987 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotSize3snapshot C_M_SNAPSHOT_TABLE on table testSnapshotSize3 2023-07-12 19:40:36,988 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:36,988 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:36,988 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=SHARED 2023-07-12 19:40:36,989 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=42, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=EXCLUSIVE 2023-07-12 19:40:36,990 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=EXCLUSIVE in 5 msec 2023-07-12 19:40:36,990 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=43, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=SHARED 2023-07-12 19:40:36,991 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=43, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=SHARED; timeout=600000, timestamp=1689191436991 2023-07-12 19:40:36,998 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotSize3snapshot 2023-07-12 19:40:36,998 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotSize3snapshot' 2023-07-12 19:40:36,999 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:36,999 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotSize3snapshot' starting 'acquire' 2023-07-12 19:40:36,999 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotSize3snapshot', kicking off acquire phase on members. 2023-07-12 19:40:36,999 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,000 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotSize3snapshot 2023-07-12 19:40:37,001 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:37,001 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:37,001 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,001 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotSize3snapshot 2023-07-12 19:40:37,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 63 2023-07-12 19:40:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotSize3snapshot 2023-07-12 19:40:37,002 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotSize3snapshot from table testSnapshotSize3 type SKIPFLUSH 2023-07-12 19:40:37,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotSize3snapshot 2023-07-12 19:40:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotSize3snapshot' with timeout 300000ms 2023-07-12 19:40:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:37,003 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotSize3snapshot' starting 'acquire' stage 2023-07-12 19:40:37,003 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotSize3snapshot' locally acquired 2023-07-12 19:40:37,003 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotSize3snapshot) in zk 2023-07-12 19:40:37,005 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,005 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotSize3snapshot 2023-07-12 19:40:37,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:37,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:37,005 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotSize3snapshot 2023-07-12 19:40:37,005 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotSize3snapshot' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:37,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3snapshot 2023-07-12 19:40:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:37,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotSize3snapshot' on coordinator 2023-07-12 19:40:37,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6879194a[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:37,007 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotSize3snapshot' starting 'in-barrier' execution. 2023-07-12 19:40:37,007 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotSize3snapshot 2023-07-12 19:40:37,008 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotSize3snapshot 2023-07-12 19:40:37,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotSize3snapshot 2023-07-12 19:40:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotSize3snapshot 2023-07-12 19:40:37,008 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotSize3snapshot' received 'reached' from coordinator. 2023-07-12 19:40:37,008 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,008 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:37,008 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:37,009 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 5 regions 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83.' region-info for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.SnapshotManifest(238): Storing 'testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615.' region-info for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,009 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/d0115a6cb0a849cea1235a2f7a35f53e] hfiles 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.SnapshotManifest(238): Storing 'testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c.' region-info for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/d0115a6cb0a849cea1235a2f7a35f53e for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/92ccedf11cef4cf884248c39062924a2] hfiles 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/92ccedf11cef4cf884248c39062924a2 for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,009 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:37,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. completed. 2023-07-12 19:40:37,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:37,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:37,021 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/5 local region snapshots. 2023-07-12 19:40:37,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:37,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.SnapshotManifest(238): Storing 'testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07.' region-info for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:37,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/f1ee2f9dba5847a2ab1da179375d8ba1] hfiles 2023-07-12 19:40:37,021 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/f1ee2f9dba5847a2ab1da179375d8ba1 for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,023 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. completed. 2023-07-12 19:40:37,023 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:37,023 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:37,023 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 2/5 local region snapshots. 2023-07-12 19:40:37,023 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:37,023 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.SnapshotManifest(238): Storing 'testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae.' region-info for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,023 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:37,023 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/c68e1caa97f044da8a80480dc854370a] hfiles 2023-07-12 19:40:37,023 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/c68e1caa97f044da8a80480dc854370a for snapshot=testSnapshotSize3snapshot 2023-07-12 19:40:37,037 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. completed. 2023-07-12 19:40:37,037 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:37,037 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 3/5 local region snapshots. 2023-07-12 19:40:37,088 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:37,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:37,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:37,090 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:37,290 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:37,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:37,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:37,292 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:37,424 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. completed. 2023-07-12 19:40:37,424 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:37,425 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 4/5 local region snapshots. 2023-07-12 19:40:37,437 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. completed. 2023-07-12 19:40:37,437 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 5/5 local region snapshots. 2023-07-12 19:40:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 5 local region snapshots. 2023-07-12 19:40:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotSize3snapshot' locally completed 2023-07-12 19:40:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotSize3snapshot' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:37,440 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotSize3snapshot' has notified controller of completion 2023-07-12 19:40:37,440 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,440 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:37,440 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotSize3snapshot' completed. 2023-07-12 19:40:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3snapshot 2023-07-12 19:40:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3snapshot 2023-07-12 19:40:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotSize3snapshot' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:37,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotSize3snapshot', counting down latch. Waiting for 0 more 2023-07-12 19:40:37,447 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotSize3snapshot' execution completed 2023-07-12 19:40:37,447 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:37,447 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:37,447 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotSize3snapshot 2023-07-12 19:40:37,447 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotSize3snapshotincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:37,449 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,449 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:37,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,449 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:37,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:37,449 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3snapshot 2023-07-12 19:40:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3snapshot 2023-07-12 19:40:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,451 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3snapshot 2023-07-12 19:40:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,455 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:37,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:37,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:37,455 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:37,455 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotSize3snapshot 2023-07-12 19:40:37,455 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotSize3snapshot 2023-07-12 19:40:37,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/testSnapshotSize3snapshot because node does not exist (not an error) 2023-07-12 19:40:37,455 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,455 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:37,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:37,456 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotSize3snapshot 2023-07-12 19:40:37,456 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotSize3snapshot 2023-07-12 19:40:37,457 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotSize3snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:37,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:37,457 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotSize3snapshot 2023-07-12 19:40:37,457 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotSize3snapshot 2023-07-12 19:40:37,457 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotSize3snapshot 2023-07-12 19:40:37,457 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotSize3snapshot 2023-07-12 19:40:37,592 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:37,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:37,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:37,594 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:37,895 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotSize3snapshot to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotSize3snapshot 2023-07-12 19:40:37,912 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x212e4f2c to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:37,916 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@443e4a3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:37,918 DEBUG [hconnection-0x354d3f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:37,919 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:37,964 DEBUG [master/jenkins-hbase4:0.Chore.1] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotSize3snapshot 2023-07-12 19:40:37,964 ERROR [master/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(166): Caught error java.lang.NullPointerException at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.getStoreFilesFromSnapshot(FileArchiverNotifierImpl.java:488) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeSnapshotSizes(FileArchiverNotifierImpl.java:404) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeAndStoreSnapshotSizes(FileArchiverNotifierImpl.java:334) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.computeSnapshotSizes(SnapshotQuotaObserverChore.java:225) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:118) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:38,094 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:38,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:38,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:38,096 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:38,719 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotSize3snapshot of table testSnapshotSize3 completed 2023-07-12 19:40:38,719 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotSize3snapshot 2023-07-12 19:40:38,719 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotSize3snapshot 2023-07-12 19:40:38,719 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotSize3snapshot C_M_SNAPSHOT_TABLE on table testSnapshotSize3 at 1689190836987Consolidate snapshot: testSnapshotSize3snapshot at 1689190837455 (+468 ms)Loading Region manifests for testSnapshotSize3snapshot at 1689190837456 (+1 ms)Writing data manifest for testSnapshotSize3snapshot at 1689190837472 (+16 ms)Verifying snapshot: testSnapshotSize3snapshot at 1689190837885 (+413 ms)Snapshot testSnapshotSize3snapshot of table testSnapshotSize3 completed at 1689190838719 (+834 ms) 2023-07-12 19:40:38,721 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=43, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=SHARED 2023-07-12 19:40:38,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotSize3, type=SHARED in 1.7340 sec 2023-07-12 19:40:38,912 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x178b38e6 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:38,917 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d059af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:38,919 DEBUG [hconnection-0x7b2ab63f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:38,920 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50624, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:39,096 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:39,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:39,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotSize3snapshot table=testSnapshotSize3 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:39,099 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x48cfac28 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:39,108 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2adba079, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:39,110 DEBUG [hconnection-0x5d6133c9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:39,113 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50630, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:39,303 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 19:40:39,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(936): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-07-12 19:40:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(143): Procedure testSnapshotSize3 was in running list but was completed. Accepting new attempt. 2023-07-12 19:40:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotSize3 2023-07-12 19:40:39,312 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotSize3' 2023-07-12 19:40:39,312 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:39,313 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotSize3' starting 'acquire' 2023-07-12 19:40:39,313 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotSize3', kicking off acquire phase on members. 2023-07-12 19:40:39,313 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,313 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,315 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:39,315 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:39,316 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,316 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 19:40:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSnapshotSize3 2023-07-12 19:40:39,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSnapshotSize3 is still present, removing 2023-07-12 19:40:39,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotSize3 2023-07-12 19:40:39,317 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotSize3' with timeout 60000ms 2023-07-12 19:40:39,317 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:39,317 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotSize3' starting 'acquire' stage 2023-07-12 19:40:39,318 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:39,318 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. started... 2023-07-12 19:40:39,318 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9869f769c7045fb07f1013a482eb9f83 1/1 column families, dataSize=84.54 KB heapSize=107.83 KB 2023-07-12 19:40:39,318 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:39,318 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 5 regions 2023-07-12 19:40:39,319 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 19:40:39,319 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. started... 2023-07-12 19:40:39,319 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. started... 2023-07-12 19:40:39,319 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 0c1b9295d50da7c1350cece1d5853d2c: 2023-07-12 19:40:39,319 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:39,319 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing b7f092ed0d4ff500eb86950c457f0615 1/1 column families, dataSize=85.37 KB heapSize=108.88 KB 2023-07-12 19:40:39,319 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:39,319 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. started... 2023-07-12 19:40:39,319 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/5 local region flush tasks. 2023-07-12 19:40:39,319 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2745): Flushing 4f6eda388e14a12df47ea1363fceed07 1/1 column families, dataSize=84.82 KB heapSize=108.18 KB 2023-07-12 19:40:39,345 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=84.54 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/.tmp/f1/91dba1ad64e74a048ef7559e5d69018d 2023-07-12 19:40:39,346 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=84.82 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/.tmp/f1/65a91a1dcec54d7e82b273a8125c0107 2023-07-12 19:40:39,349 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=85.37 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/.tmp/f1/b32e7c86a5104ffa856230560ad18c75 2023-07-12 19:40:39,354 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/.tmp/f1/91dba1ad64e74a048ef7559e5d69018d as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/91dba1ad64e74a048ef7559e5d69018d 2023-07-12 19:40:39,356 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/.tmp/f1/65a91a1dcec54d7e82b273a8125c0107 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/65a91a1dcec54d7e82b273a8125c0107 2023-07-12 19:40:39,362 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/91dba1ad64e74a048ef7559e5d69018d, entries=306, sequenceid=48, filesize=90.1 K 2023-07-12 19:40:39,363 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/.tmp/f1/b32e7c86a5104ffa856230560ad18c75 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/b32e7c86a5104ffa856230560ad18c75 2023-07-12 19:40:39,363 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~84.54 KB/86571, heapSize ~107.81 KB/110400, currentSize=0 B/0 for 9869f769c7045fb07f1013a482eb9f83 in 45ms, sequenceid=48, compaction requested=false 2023-07-12 19:40:39,364 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9869f769c7045fb07f1013a482eb9f83: 2023-07-12 19:40:39,364 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:39,364 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:39,364 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. started... 2023-07-12 19:40:39,364 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2eb66a3eec48e0be370ce75cea0caeae 1/1 column families, dataSize=28.18 KB heapSize=36.11 KB 2023-07-12 19:40:39,364 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 2/5 local region flush tasks. 2023-07-12 19:40:39,366 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/65a91a1dcec54d7e82b273a8125c0107, entries=307, sequenceid=50, filesize=90.4 K 2023-07-12 19:40:39,367 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2948): Finished flush of dataSize ~84.82 KB/86855, heapSize ~108.16 KB/110760, currentSize=0 B/0 for 4f6eda388e14a12df47ea1363fceed07 in 48ms, sequenceid=50, compaction requested=false 2023-07-12 19:40:39,368 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 4f6eda388e14a12df47ea1363fceed07: 2023-07-12 19:40:39,368 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:39,368 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 3/5 local region flush tasks. 2023-07-12 19:40:39,370 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/b32e7c86a5104ffa856230560ad18c75, entries=309, sequenceid=50, filesize=91.0 K 2023-07-12 19:40:39,371 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~85.37 KB/87423, heapSize ~108.87 KB/111480, currentSize=0 B/0 for b7f092ed0d4ff500eb86950c457f0615 in 52ms, sequenceid=50, compaction requested=false 2023-07-12 19:40:39,371 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for b7f092ed0d4ff500eb86950c457f0615: 2023-07-12 19:40:39,371 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:39,371 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 4/5 local region flush tasks. 2023-07-12 19:40:39,783 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=28.18 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/.tmp/f1/393e97df53ee41108d002097fc89c36c 2023-07-12 19:40:39,792 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/.tmp/f1/393e97df53ee41108d002097fc89c36c as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/393e97df53ee41108d002097fc89c36c 2023-07-12 19:40:39,799 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/393e97df53ee41108d002097fc89c36c, entries=102, sequenceid=48, filesize=33.1 K 2023-07-12 19:40:39,800 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~28.18 KB/28857, heapSize ~36.09 KB/36960, currentSize=0 B/0 for 2eb66a3eec48e0be370ce75cea0caeae in 436ms, sequenceid=48, compaction requested=false 2023-07-12 19:40:39,800 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2eb66a3eec48e0be370ce75cea0caeae: 2023-07-12 19:40:39,800 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:39,800 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 5/5 local region flush tasks. 2023-07-12 19:40:39,800 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 5 local region flush tasks. 2023-07-12 19:40:39,800 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,800 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotSize3' locally acquired 2023-07-12 19:40:39,800 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotSize3) in zk 2023-07-12 19:40:39,802 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,802 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:39,802 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,802 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotSize3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:39,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotSize3' on coordinator 2023-07-12 19:40:39,804 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotSize3' starting 'in-barrier' execution. 2023-07-12 19:40:39,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ff7414[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:39,805 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,806 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,806 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotSize3' received 'reached' from coordinator. 2023-07-12 19:40:39,807 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotSize3' locally completed 2023-07-12 19:40:39,807 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,807 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:39,807 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotSize3' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:39,809 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,809 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotSize3' has notified controller of completion 2023-07-12 19:40:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:39,809 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotSize3' completed. 2023-07-12 19:40:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotSize3' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:39,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotSize3', counting down latch. Waiting for 0 more 2023-07-12 19:40:39,818 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotSize3' execution completed 2023-07-12 19:40:39,818 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:39,818 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:39,818 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotSize3 2023-07-12 19:40:39,818 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotSize3including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 19:40:39,820 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,820 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,820 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:39,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:39,821 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:39,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,823 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:39,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,826 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,826 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,826 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:39,826 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:39,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:39,827 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,827 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,827 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,827 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:39,827 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSnapshotSize3' 2023-07-12 19:40:39,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:39,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 19:40:39,827 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:39,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSnapshotSize3 because node does not exist (not an error) 2023-07-12 19:40:39,827 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSnapshotSize3'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:39,828 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 19:40:39,915 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x72a18d5b to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:39,920 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fb383b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:39,922 DEBUG [hconnection-0x21220996-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:39,923 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50646, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:39,928 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 19:40:39,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 19:40:39,929 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 19:40:39,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(936): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-07-12 19:40:39,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(143): Procedure testSnapshotSize3 was in running list but was completed. Accepting new attempt. 2023-07-12 19:40:39,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotSize3 2023-07-12 19:40:39,936 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotSize3' 2023-07-12 19:40:39,936 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:39,937 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotSize3' starting 'acquire' 2023-07-12 19:40:39,937 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotSize3', kicking off acquire phase on members. 2023-07-12 19:40:39,937 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,937 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,940 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:39,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 19:40:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSnapshotSize3 2023-07-12 19:40:39,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSnapshotSize3 is still present, removing 2023-07-12 19:40:39,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotSize3 2023-07-12 19:40:39,942 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotSize3' with timeout 60000ms 2023-07-12 19:40:39,942 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:39,944 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotSize3' starting 'acquire' stage 2023-07-12 19:40:39,951 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,951 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:39,951 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 5 regions 2023-07-12 19:40:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 19:40:39,952 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. started... 2023-07-12 19:40:39,952 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9869f769c7045fb07f1013a482eb9f83: 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. started... 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4f6eda388e14a12df47ea1363fceed07: 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. started... 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2eb66a3eec48e0be370ce75cea0caeae: 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:39,952 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:39,952 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. started... 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. started... 2023-07-12 19:40:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/5 local region flush tasks. 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 0c1b9295d50da7c1350cece1d5853d2c: 2023-07-12 19:40:39,953 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 2/5 local region flush tasks. 2023-07-12 19:40:39,953 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:40:39,954 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for b7f092ed0d4ff500eb86950c457f0615: 2023-07-12 19:40:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 3/5 local region flush tasks. 2023-07-12 19:40:39,954 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 4/5 local region flush tasks. 2023-07-12 19:40:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 5/5 local region flush tasks. 2023-07-12 19:40:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 5 local region flush tasks. 2023-07-12 19:40:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotSize3' locally acquired 2023-07-12 19:40:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotSize3) in zk 2023-07-12 19:40:39,954 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,955 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:39,956 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,956 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:39,958 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,958 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotSize3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:39,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotSize3' on coordinator 2023-07-12 19:40:39,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7466600e[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:39,960 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotSize3' starting 'in-barrier' execution. 2023-07-12 19:40:39,960 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,961 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotSize3' received 'reached' from coordinator. 2023-07-12 19:40:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotSize3' locally completed 2023-07-12 19:40:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotSize3' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:39,962 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,962 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:39,963 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:39,963 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotSize3' has notified controller of completion 2023-07-12 19:40:39,963 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:39,963 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotSize3' completed. 2023-07-12 19:40:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:39,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotSize3' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:39,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotSize3', counting down latch. Waiting for 0 more 2023-07-12 19:40:39,967 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotSize3' execution completed 2023-07-12 19:40:39,967 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:39,967 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:39,967 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotSize3 2023-07-12 19:40:39,967 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotSize3including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 19:40:39,970 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,970 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:39,970 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:39,971 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,973 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotSize3 2023-07-12 19:40:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,977 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:39,977 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,977 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:39,977 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:39,977 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:39,977 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSnapshotSize3 2023-07-12 19:40:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:39,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSnapshotSize3' 2023-07-12 19:40:39,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 19:40:39,978 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:39,978 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,978 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSnapshotSize3 2023-07-12 19:40:39,978 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSnapshotSize3 2023-07-12 19:40:39,979 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSnapshotSize3'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:39,979 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 19:40:39,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:39,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSnapshotSize3 because node does not exist (not an error) 2023-07-12 19:40:40,079 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 19:40:40,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 19:40:40,083 DEBUG [Listener at localhost/32953] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 19:40:40,085 DEBUG [Listener at localhost/32953] regionserver.HStore(1912): 9869f769c7045fb07f1013a482eb9f83/f1 is initiating major compaction (all files) 2023-07-12 19:40:40,085 INFO [Listener at localhost/32953] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:40:40,085 INFO [Listener at localhost/32953] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:40,086 INFO [Listener at localhost/32953] regionserver.HRegion(2259): Starting compaction of 9869f769c7045fb07f1013a482eb9f83/f1 in testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:40:40,086 INFO [Listener at localhost/32953] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/d0115a6cb0a849cea1235a2f7a35f53e, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/91dba1ad64e74a048ef7559e5d69018d] into tmpdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/.tmp, totalSize=180.3 K 2023-07-12 19:40:40,087 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting d0115a6cb0a849cea1235a2f7a35f53e, keycount=306, bloomtype=ROW, size=90.1 K, encoding=NONE, compression=NONE, seqNum=24, earliestPutTs=1689190835333 2023-07-12 19:40:40,088 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting 91dba1ad64e74a048ef7559e5d69018d, keycount=306, bloomtype=ROW, size=90.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1689190839173 2023-07-12 19:40:40,124 INFO [Listener at localhost/32953] throttle.PressureAwareThroughputController(145): 9869f769c7045fb07f1013a482eb9f83#f1#compaction#8 average throughput is 5.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 19:40:40,334 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 19:40:40,536 DEBUG [Listener at localhost/32953] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/.tmp/f1/c096ebcfc2cf4bc8996561e527408d17 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/c096ebcfc2cf4bc8996561e527408d17 2023-07-12 19:40:40,550 INFO [Listener at localhost/32953] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 9869f769c7045fb07f1013a482eb9f83/f1 of 9869f769c7045fb07f1013a482eb9f83 into c096ebcfc2cf4bc8996561e527408d17(size=90.2 K), total size for store is 90.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 19:40:40,550 DEBUG [Listener at localhost/32953] regionserver.HRegion(2289): Compaction status journal for 9869f769c7045fb07f1013a482eb9f83: 2023-07-12 19:40:40,550 DEBUG [Listener at localhost/32953] compactions.SortedCompactionPolicy(75): Selecting compaction from 0 store files, 0 compacting, 0 eligible, 16 blocking 2023-07-12 19:40:40,550 DEBUG [Listener at localhost/32953] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 19:40:40,550 DEBUG [Listener at localhost/32953] regionserver.HStore(1912): b7f092ed0d4ff500eb86950c457f0615/f1 is initiating major compaction (all files) 2023-07-12 19:40:40,551 INFO [Listener at localhost/32953] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:40:40,551 INFO [Listener at localhost/32953] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:40,551 INFO [Listener at localhost/32953] regionserver.HRegion(2259): Starting compaction of b7f092ed0d4ff500eb86950c457f0615/f1 in testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:40:40,551 INFO [Listener at localhost/32953] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/92ccedf11cef4cf884248c39062924a2, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/b32e7c86a5104ffa856230560ad18c75] into tmpdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/.tmp, totalSize=181.9 K 2023-07-12 19:40:40,552 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting 92ccedf11cef4cf884248c39062924a2, keycount=309, bloomtype=ROW, size=91.0 K, encoding=NONE, compression=NONE, seqNum=25, earliestPutTs=1689190835328 2023-07-12 19:40:40,553 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting b32e7c86a5104ffa856230560ad18c75, keycount=309, bloomtype=ROW, size=91.0 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1689190839171 2023-07-12 19:40:40,570 INFO [Listener at localhost/32953] throttle.PressureAwareThroughputController(145): b7f092ed0d4ff500eb86950c457f0615#f1#compaction#9 average throughput is 11.91 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 19:40:40,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7a623f22 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:40,914 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22111f5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:40,915 DEBUG [hconnection-0x67b384c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:40,917 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:40,967 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testSnapshotSize3 moved into observance of table space quota. 2023-07-12 19:40:40,981 DEBUG [Listener at localhost/32953] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/.tmp/f1/bbbb49fb7f9f4f7388b2ada987951aaf as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/bbbb49fb7f9f4f7388b2ada987951aaf 2023-07-12 19:40:40,988 INFO [Listener at localhost/32953] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in b7f092ed0d4ff500eb86950c457f0615/f1 of b7f092ed0d4ff500eb86950c457f0615 into bbbb49fb7f9f4f7388b2ada987951aaf(size=91.0 K), total size for store is 91.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 19:40:40,988 DEBUG [Listener at localhost/32953] regionserver.HRegion(2289): Compaction status journal for b7f092ed0d4ff500eb86950c457f0615: 2023-07-12 19:40:40,988 DEBUG [Listener at localhost/32953] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 19:40:40,988 DEBUG [Listener at localhost/32953] regionserver.HStore(1912): 4f6eda388e14a12df47ea1363fceed07/f1 is initiating major compaction (all files) 2023-07-12 19:40:40,988 INFO [Listener at localhost/32953] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:40:40,988 INFO [Listener at localhost/32953] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:40,989 INFO [Listener at localhost/32953] regionserver.HRegion(2259): Starting compaction of 4f6eda388e14a12df47ea1363fceed07/f1 in testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:40:40,989 INFO [Listener at localhost/32953] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/f1ee2f9dba5847a2ab1da179375d8ba1, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/65a91a1dcec54d7e82b273a8125c0107] into tmpdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/.tmp, totalSize=180.8 K 2023-07-12 19:40:40,990 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting f1ee2f9dba5847a2ab1da179375d8ba1, keycount=307, bloomtype=ROW, size=90.4 K, encoding=NONE, compression=NONE, seqNum=25, earliestPutTs=1689190835331 2023-07-12 19:40:40,991 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting 65a91a1dcec54d7e82b273a8125c0107, keycount=307, bloomtype=ROW, size=90.4 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1689190839172 2023-07-12 19:40:41,005 INFO [Listener at localhost/32953] throttle.PressureAwareThroughputController(145): 4f6eda388e14a12df47ea1363fceed07#f1#compaction#10 average throughput is 20.71 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 19:40:41,020 DEBUG [Listener at localhost/32953] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/.tmp/f1/657868046557471cbf21d8d9789a9014 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/657868046557471cbf21d8d9789a9014 2023-07-12 19:40:41,027 INFO [Listener at localhost/32953] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 4f6eda388e14a12df47ea1363fceed07/f1 of 4f6eda388e14a12df47ea1363fceed07 into 657868046557471cbf21d8d9789a9014(size=90.5 K), total size for store is 90.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 19:40:41,028 DEBUG [Listener at localhost/32953] regionserver.HRegion(2289): Compaction status journal for 4f6eda388e14a12df47ea1363fceed07: 2023-07-12 19:40:41,028 DEBUG [Listener at localhost/32953] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 19:40:41,028 DEBUG [Listener at localhost/32953] regionserver.HStore(1912): 2eb66a3eec48e0be370ce75cea0caeae/f1 is initiating major compaction (all files) 2023-07-12 19:40:41,028 INFO [Listener at localhost/32953] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:40:41,028 INFO [Listener at localhost/32953] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:40:41,029 INFO [Listener at localhost/32953] regionserver.HRegion(2259): Starting compaction of 2eb66a3eec48e0be370ce75cea0caeae/f1 in testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:40:41,029 INFO [Listener at localhost/32953] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/c68e1caa97f044da8a80480dc854370a, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/393e97df53ee41108d002097fc89c36c] into tmpdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/.tmp, totalSize=66.2 K 2023-07-12 19:40:41,029 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting c68e1caa97f044da8a80480dc854370a, keycount=102, bloomtype=ROW, size=33.1 K, encoding=NONE, compression=NONE, seqNum=24, earliestPutTs=1689190835334 2023-07-12 19:40:41,030 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting 393e97df53ee41108d002097fc89c36c, keycount=102, bloomtype=ROW, size=33.1 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1689190839174 2023-07-12 19:40:41,042 INFO [Listener at localhost/32953] throttle.PressureAwareThroughputController(145): 2eb66a3eec48e0be370ce75cea0caeae#f1#compaction#11 average throughput is 9.17 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 19:40:41,454 DEBUG [Listener at localhost/32953] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/.tmp/f1/b17a9be568dd4b68975c58bc747f64c5 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/b17a9be568dd4b68975c58bc747f64c5 2023-07-12 19:40:41,461 INFO [Listener at localhost/32953] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 2eb66a3eec48e0be370ce75cea0caeae/f1 of 2eb66a3eec48e0be370ce75cea0caeae into b17a9be568dd4b68975c58bc747f64c5(size=33.2 K), total size for store is 33.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 19:40:41,461 DEBUG [Listener at localhost/32953] regionserver.HRegion(2289): Compaction status journal for 2eb66a3eec48e0be370ce75cea0caeae: 2023-07-12 19:40:41,463 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:40:41,465 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testSnapshotSize3: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=609.28 KB/1 GB] 2023-07-12 19:40:41,465 DEBUG [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$2(261): Current usage=623898 snapshotSize=311949 2023-07-12 19:40:41,465 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:40:41,466 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:41,566 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:41,666 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:41,766 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:41,867 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:41,967 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:41,989 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testSnapshotSize3 moved into observance of table space quota. 2023-07-12 19:40:41,990 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7ae3d49f to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:41,994 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4dfe9692, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:41,997 DEBUG [hconnection-0x4e1f0852-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:41,999 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:42,067 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,168 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,268 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,368 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,469 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,569 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,669 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,770 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,870 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x0eead69d to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:42,915 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@148067f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:42,917 DEBUG [hconnection-0x193a77a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:42,918 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:42,955 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/d0115a6cb0a849cea1235a2f7a35f53e, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/91dba1ad64e74a048ef7559e5d69018d] to archive 2023-07-12 19:40:42,956 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 19:40:42,961 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/d0115a6cb0a849cea1235a2f7a35f53e to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/d0115a6cb0a849cea1235a2f7a35f53e 2023-07-12 19:40:42,963 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/91dba1ad64e74a048ef7559e5d69018d to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/f1/91dba1ad64e74a048ef7559e5d69018d 2023-07-12 19:40:42,970 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$NoFilesToDischarge(765): 9869f769c7045fb07f1013a482eb9f83 still has compacted files 2023-07-12 19:40:42,972 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testSnapshotSize3 moved into observance of table space quota. 2023-07-12 19:40:42,994 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/92ccedf11cef4cf884248c39062924a2, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/b32e7c86a5104ffa856230560ad18c75] to archive 2023-07-12 19:40:42,995 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 19:40:42,997 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/92ccedf11cef4cf884248c39062924a2 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/92ccedf11cef4cf884248c39062924a2 2023-07-12 19:40:42,998 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/b32e7c86a5104ffa856230560ad18c75 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/f1/b32e7c86a5104ffa856230560ad18c75 2023-07-12 19:40:43,020 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d23dcc6 to 127.0.0.1:57973 2023-07-12 19:40:43,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36499595 to 127.0.0.1:57973 2023-07-12 19:40:43,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a46a97a to 127.0.0.1:57973 2023-07-12 19:40:43,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40568a6c to 127.0.0.1:57973 2023-07-12 19:40:43,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f444882 to 127.0.0.1:57973 2023-07-12 19:40:43,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4be392cb to 127.0.0.1:57973 2023-07-12 19:40:43,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6770eaac to 127.0.0.1:57973 2023-07-12 19:40:43,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b19cf23 to 127.0.0.1:57973 2023-07-12 19:40:43,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70554738 to 127.0.0.1:57973 2023-07-12 19:40:43,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x442f6434 to 127.0.0.1:57973 2023-07-12 19:40:43,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7211b91b to 127.0.0.1:57973 2023-07-12 19:40:43,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,023 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5459642a to 127.0.0.1:57973 2023-07-12 19:40:43,023 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,023 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f5a0c0d to 127.0.0.1:57973 2023-07-12 19:40:43,023 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,023 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x649ff042 to 127.0.0.1:57973 2023-07-12 19:40:43,023 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ab12cb3 to 127.0.0.1:57973 2023-07-12 19:40:43,024 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21689cdf to 127.0.0.1:57973 2023-07-12 19:40:43,024 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1abe42a0 to 127.0.0.1:57973 2023-07-12 19:40:43,024 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20750642 to 127.0.0.1:57973 2023-07-12 19:40:43,025 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,025 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eead69d to 127.0.0.1:57973 2023-07-12 19:40:43,025 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,025 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ae3d49f to 127.0.0.1:57973 2023-07-12 19:40:43,025 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,026 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a623f22 to 127.0.0.1:57973 2023-07-12 19:40:43,026 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,026 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7792e9a5 to 127.0.0.1:57973 2023-07-12 19:40:43,027 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,027 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c1d91f6 to 127.0.0.1:57973 2023-07-12 19:40:43,027 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,029 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72a18d5b to 127.0.0.1:57973 2023-07-12 19:40:43,029 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,029 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48cfac28 to 127.0.0.1:57973 2023-07-12 19:40:43,029 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,030 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x178b38e6 to 127.0.0.1:57973 2023-07-12 19:40:43,030 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,030 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x212e4f2c to 127.0.0.1:57973 2023-07-12 19:40:43,030 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,034 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/f1ee2f9dba5847a2ab1da179375d8ba1, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/65a91a1dcec54d7e82b273a8125c0107] to archive 2023-07-12 19:40:43,035 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 19:40:43,037 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/f1ee2f9dba5847a2ab1da179375d8ba1 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/f1ee2f9dba5847a2ab1da179375d8ba1 2023-07-12 19:40:43,039 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/65a91a1dcec54d7e82b273a8125c0107 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/f1/65a91a1dcec54d7e82b273a8125c0107 2023-07-12 19:40:43,048 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/c68e1caa97f044da8a80480dc854370a, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/393e97df53ee41108d002097fc89c36c] to archive 2023-07-12 19:40:43,049 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 19:40:43,051 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/c68e1caa97f044da8a80480dc854370a to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/c68e1caa97f044da8a80480dc854370a 2023-07-12 19:40:43,052 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/393e97df53ee41108d002097fc89c36c to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/f1/393e97df53ee41108d002097fc89c36c 2023-07-12 19:40:43,071 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x776809ad to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:43,075 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ef57edc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:43,331 DEBUG [hconnection-0x25752c91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:43,335 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:43,418 INFO [Listener at localhost/32953] hbase.ResourceChecker(175): after: quotas.TestSnapshotQuotaObserverChore#testSnapshotSize Thread=310 (was 283) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7ae3d49f-SendThread(127.0.0.1:57973) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1261) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x178b38e6 java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1257) java.lang.Thread.join(Thread.java:1331) org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1455) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1485) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x25752c91-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x178b38e6-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x776809ad sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1215859492_17 at /127.0.0.1:40176 [Waiting for operation #23] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x212e4f2c java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1257) java.lang.Thread.join(Thread.java:1331) org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1455) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1485) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x212e4f2c-SendThread(127.0.0.1:57973) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1261) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7ae3d49f-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x20750642-SendThread(127.0.0.1:57973) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1261) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-361464297_17 at /127.0.0.1:42022 [Waiting for operation #24] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-361464297_17 at /127.0.0.1:40174 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7c1d91f6-SendThread(127.0.0.1:57973) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1261) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x212e4f2c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1215859492_17 at /127.0.0.1:42000 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7c1d91f6 java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1257) java.lang.Thread.join(Thread.java:1331) org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1455) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1485) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x72a18d5b-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x59c16f00-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x20750642-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x178b38e6-SendThread(127.0.0.1:57973) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1261) Potentially hanging thread: hconnection-0x59c16f00-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x25752c91-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7c1d91f6-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-361464297_17 at /127.0.0.1:40098 [Waiting for operation #19] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x72a18d5b-SendThread(127.0.0.1:57973) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1261) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x72a18d5b java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1257) java.lang.Thread.join(Thread.java:1331) org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1455) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1485) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x7ae3d49f java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1257) java.lang.Thread.join(Thread.java:1331) org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1455) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1485) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x20750642 java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1257) java.lang.Thread.join(Thread.java:1331) org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1455) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1485) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1215859492_17 at /127.0.0.1:41984 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x59c16f00-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x48cfac28-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-361464297_17 at /127.0.0.1:40188 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x48cfac28-SendThread(127.0.0.1:57973) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1261) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-361464297_17 at /127.0.0.1:40190 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x776809ad-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x48cfac28 java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1257) java.lang.Thread.join(Thread.java:1331) org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1455) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1485) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1861773656.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:57973@0x776809ad-SendThread(127.0.0.1:57973) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1215859492_17 at /127.0.0.1:40282 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=617 (was 592) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=96 (was 104), ProcessCount=172 (was 172), AvailableMemoryMB=8143 (was 8022) - AvailableMemoryMB LEAK? - 2023-07-12 19:40:43,433 INFO [Listener at localhost/32953] hbase.ResourceChecker(147): before: quotas.TestSnapshotQuotaObserverChore#testSnapshotsFromNamespaces Thread=310, OpenFileDescriptor=617, MaxFileDescriptor=60000, SystemLoadAverage=96, ProcessCount=172, AvailableMemoryMB=8142 2023-07-12 19:40:43,436 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x1536ceeb to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:43,441 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64a62a96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:43,443 DEBUG [hconnection-0x6f8fb390-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:43,446 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:43,450 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(447): Deleting quota for table: testSnapshotSize3 2023-07-12 19:40:43,452 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1536ceeb to 127.0.0.1:57973 2023-07-12 19:40:43,452 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:43,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$15(3014): Client=jenkins//172.31.14.131 creating {NAME => 'snapshots_from_namespaces'} 2023-07-12 19:40:43,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=44, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=snapshots_from_namespaces 2023-07-12 19:40:43,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=44 2023-07-12 19:40:43,466 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:40:43,469 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, state=SUCCESS; CreateNamespaceProcedure, namespace=snapshots_from_namespaces in 10 msec 2023-07-12 19:40:43,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=44 2023-07-12 19:40:43,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'snapshots_from_namespaces:testSnapshotsFromNamespaces4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:43,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=45, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 2023-07-12 19:40:43,571 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=45, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:43,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "snapshots_from_namespaces" qualifier: "testSnapshotsFromNamespaces4" procId is: 45 2023-07-12 19:40:43,572 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=45, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:43,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=45 2023-07-12 19:40:43,574 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,574 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2 empty. 2023-07-12 19:40:43,575 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,575 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived snapshots_from_namespaces:testSnapshotsFromNamespaces4 regions 2023-07-12 19:40:43,590 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:43,591 INFO [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3c735bb63690fff52349b93445eb2aa2, NAME => 'snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='snapshots_from_namespaces:testSnapshotsFromNamespaces4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:43,602 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(866): Instantiated snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:43,602 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(1604): Closing 3c735bb63690fff52349b93445eb2aa2, disabling compactions & flushes 2023-07-12 19:40:43,602 INFO [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(1626): Closing region snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:43,602 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:43,602 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(1714): Acquired close lock on snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. after waiting 0 ms 2023-07-12 19:40:43,602 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(1724): Updates disabled for region snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:43,602 INFO [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(1838): Closed snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:43,602 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces4-pool-0] regionserver.HRegion(1558): Region close journal for 3c735bb63690fff52349b93445eb2aa2: 2023-07-12 19:40:43,606 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=45, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:43,607 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2.","families":{"info":[{"qualifier":"regioninfo","vlen":80,"tag":[],"timestamp":"1689190843607"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190843607"}]},"ts":"1689190843607"} 2023-07-12 19:40:43,609 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:43,610 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=45, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:43,610 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190843610"}]},"ts":"1689190843610"} 2023-07-12 19:40:43,611 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, state=ENABLING in hbase:meta 2023-07-12 19:40:43,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4, region=3c735bb63690fff52349b93445eb2aa2, ASSIGN}] 2023-07-12 19:40:43,624 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=46, ppid=45, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4, region=3c735bb63690fff52349b93445eb2aa2, ASSIGN 2023-07-12 19:40:43,625 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=46, ppid=45, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4, region=3c735bb63690fff52349b93445eb2aa2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=45 2023-07-12 19:40:43,776 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=46 updating hbase:meta row=3c735bb63690fff52349b93445eb2aa2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:43,777 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2.","families":{"info":[{"qualifier":"regioninfo","vlen":80,"tag":[],"timestamp":"1689190843776"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190843776"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190843776"}]},"ts":"1689190843776"} 2023-07-12 19:40:43,779 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=46, state=RUNNABLE; OpenRegionProcedure 3c735bb63690fff52349b93445eb2aa2, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:43,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=45 2023-07-12 19:40:43,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:43,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3c735bb63690fff52349b93445eb2aa2, NAME => 'snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:43,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotsFromNamespaces4 3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:43,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,936 INFO [StoreOpener-3c735bb63690fff52349b93445eb2aa2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,937 DEBUG [StoreOpener-3c735bb63690fff52349b93445eb2aa2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2/f1 2023-07-12 19:40:43,937 DEBUG [StoreOpener-3c735bb63690fff52349b93445eb2aa2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2/f1 2023-07-12 19:40:43,938 INFO [StoreOpener-3c735bb63690fff52349b93445eb2aa2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3c735bb63690fff52349b93445eb2aa2 columnFamilyName f1 2023-07-12 19:40:43,938 INFO [StoreOpener-3c735bb63690fff52349b93445eb2aa2-1] regionserver.HStore(310): Store=3c735bb63690fff52349b93445eb2aa2/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:43,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:40:43,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:43,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3c735bb63690fff52349b93445eb2aa2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74653958, jitterRate=0.11243066191673279}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:43,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3c735bb63690fff52349b93445eb2aa2: 2023-07-12 19:40:43,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2., pid=47, masterSystemTime=1689190843931 2023-07-12 19:40:43,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:43,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:43,947 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=46 updating hbase:meta row=3c735bb63690fff52349b93445eb2aa2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:43,948 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2.","families":{"info":[{"qualifier":"regioninfo","vlen":80,"tag":[],"timestamp":"1689190843947"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190843947"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190843947"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190843947"}]},"ts":"1689190843947"} 2023-07-12 19:40:43,950 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=46 2023-07-12 19:40:43,950 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=46, state=SUCCESS; OpenRegionProcedure 3c735bb63690fff52349b93445eb2aa2, server=jenkins-hbase4.apache.org,44631,1689190811374 in 170 msec 2023-07-12 19:40:43,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=45 2023-07-12 19:40:43,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=45, state=SUCCESS; TransitRegionStateProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4, region=3c735bb63690fff52349b93445eb2aa2, ASSIGN in 329 msec 2023-07-12 19:40:43,953 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=45, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:43,953 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190843953"}]},"ts":"1689190843953"} 2023-07-12 19:40:43,954 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, state=ENABLED in hbase:meta 2023-07-12 19:40:43,957 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=45, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:43,958 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, state=SUCCESS; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 in 389 msec 2023-07-12 19:40:43,985 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x6e1c1974 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:43,989 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15e6d86e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:43,992 DEBUG [hconnection-0x6b48769c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:43,994 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:44,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=45 2023-07-12 19:40:44,177 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: snapshots_from_namespaces:testSnapshotsFromNamespaces4, procId: 45 completed 2023-07-12 19:40:44,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'snapshots_from_namespaces:testSnapshotsFromNamespaces5', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:44,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=48, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 2023-07-12 19:40:44,182 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=48, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:44,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "snapshots_from_namespaces" qualifier: "testSnapshotsFromNamespaces5" procId is: 48 2023-07-12 19:40:44,183 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=48, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:44,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=48 2023-07-12 19:40:44,185 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,186 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f empty. 2023-07-12 19:40:44,186 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,186 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived snapshots_from_namespaces:testSnapshotsFromNamespaces5 regions 2023-07-12 19:40:44,196 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:44,197 INFO [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(7675): creating {ENCODED => c52d439f063558619e98c674c204fb3f, NAME => 'snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='snapshots_from_namespaces:testSnapshotsFromNamespaces5', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:44,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=48 2023-07-12 19:40:44,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=48 2023-07-12 19:40:44,604 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(866): Instantiated snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:44,604 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(1604): Closing c52d439f063558619e98c674c204fb3f, disabling compactions & flushes 2023-07-12 19:40:44,604 INFO [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(1626): Closing region snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:44,604 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:44,604 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(1714): Acquired close lock on snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. after waiting 0 ms 2023-07-12 19:40:44,604 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(1724): Updates disabled for region snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:44,604 INFO [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(1838): Closed snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:44,604 DEBUG [RegionOpenAndInit-snapshots_from_namespaces:testSnapshotsFromNamespaces5-pool-0] regionserver.HRegion(1558): Region close journal for c52d439f063558619e98c674c204fb3f: 2023-07-12 19:40:44,606 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=48, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:44,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.","families":{"info":[{"qualifier":"regioninfo","vlen":80,"tag":[],"timestamp":"1689190844607"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190844607"}]},"ts":"1689190844607"} 2023-07-12 19:40:44,609 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:44,610 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=48, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:44,610 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces5","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190844610"}]},"ts":"1689190844610"} 2023-07-12 19:40:44,611 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, state=ENABLING in hbase:meta 2023-07-12 19:40:44,615 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5, region=c52d439f063558619e98c674c204fb3f, ASSIGN}] 2023-07-12 19:40:44,616 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=49, ppid=48, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5, region=c52d439f063558619e98c674c204fb3f, ASSIGN 2023-07-12 19:40:44,617 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=49, ppid=48, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5, region=c52d439f063558619e98c674c204fb3f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:44,768 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=49 updating hbase:meta row=c52d439f063558619e98c674c204fb3f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:44,768 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.","families":{"info":[{"qualifier":"regioninfo","vlen":80,"tag":[],"timestamp":"1689190844768"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190844768"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190844768"}]},"ts":"1689190844768"} 2023-07-12 19:40:44,770 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE; OpenRegionProcedure c52d439f063558619e98c674c204fb3f, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:44,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=48 2023-07-12 19:40:44,924 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:44,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c52d439f063558619e98c674c204fb3f, NAME => 'snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:44,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotsFromNamespaces5 c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:44,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,925 INFO [StoreOpener-c52d439f063558619e98c674c204fb3f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,927 DEBUG [StoreOpener-c52d439f063558619e98c674c204fb3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f/f1 2023-07-12 19:40:44,927 DEBUG [StoreOpener-c52d439f063558619e98c674c204fb3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f/f1 2023-07-12 19:40:44,927 INFO [StoreOpener-c52d439f063558619e98c674c204fb3f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c52d439f063558619e98c674c204fb3f columnFamilyName f1 2023-07-12 19:40:44,928 INFO [StoreOpener-c52d439f063558619e98c674c204fb3f-1] regionserver.HStore(310): Store=c52d439f063558619e98c674c204fb3f/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:44,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c52d439f063558619e98c674c204fb3f 2023-07-12 19:40:44,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:44,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c52d439f063558619e98c674c204fb3f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71437453, jitterRate=0.06450100243091583}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:44,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c52d439f063558619e98c674c204fb3f: 2023-07-12 19:40:44,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f., pid=50, masterSystemTime=1689190844921 2023-07-12 19:40:44,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:44,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:44,936 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=49 updating hbase:meta row=c52d439f063558619e98c674c204fb3f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:44,936 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.","families":{"info":[{"qualifier":"regioninfo","vlen":80,"tag":[],"timestamp":"1689190844936"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190844936"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190844936"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190844936"}]},"ts":"1689190844936"} 2023-07-12 19:40:44,939 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x12fd445d to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:44,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=50, resume processing ppid=49 2023-07-12 19:40:44,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; OpenRegionProcedure c52d439f063558619e98c674c204fb3f, server=jenkins-hbase4.apache.org,44631,1689190811374 in 168 msec 2023-07-12 19:40:44,941 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-07-12 19:40:44,941 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; TransitRegionStateProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5, region=c52d439f063558619e98c674c204fb3f, ASSIGN in 324 msec 2023-07-12 19:40:44,941 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=48, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:44,941 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"snapshots_from_namespaces:testSnapshotsFromNamespaces5","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190844941"}]},"ts":"1689190844941"} 2023-07-12 19:40:44,943 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, state=ENABLED in hbase:meta 2023-07-12 19:40:44,953 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12cf115f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:44,965 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=48, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:44,965 DEBUG [hconnection-0x230114a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:44,967 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:44,967 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, state=SUCCESS; CreateTableProcedure table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 in 786 msec 2023-07-12 19:40:45,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=48 2023-07-12 19:40:45,288 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: snapshots_from_namespaces:testSnapshotsFromNamespaces5, procId: 48 completed 2023-07-12 19:40:45,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testSnapshotsFromNamespaces6', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:45,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=51, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSnapshotsFromNamespaces6 2023-07-12 19:40:45,293 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=51, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSnapshotsFromNamespaces6 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:45,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testSnapshotsFromNamespaces6" procId is: 51 2023-07-12 19:40:45,294 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=51, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSnapshotsFromNamespaces6 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:45,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 19:40:45,295 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:45,296 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9 empty. 2023-07-12 19:40:45,296 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:45,296 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testSnapshotsFromNamespaces6 regions 2023-07-12 19:40:45,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 19:40:45,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 19:40:45,708 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testSnapshotsFromNamespaces6/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:45,709 INFO [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(7675): creating {ENCODED => ba20086e056a19e754e00b6bd66152a9, NAME => 'testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSnapshotsFromNamespaces6', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:45,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 19:40:45,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4ba0d5c2 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:45,910 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6767bb3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:45,912 DEBUG [hconnection-0x32dda9ce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:45,914 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:46,115 DEBUG [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(866): Instantiated testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:46,115 DEBUG [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(1604): Closing ba20086e056a19e754e00b6bd66152a9, disabling compactions & flushes 2023-07-12 19:40:46,115 INFO [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(1626): Closing region testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:46,115 DEBUG [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:46,115 DEBUG [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. after waiting 0 ms 2023-07-12 19:40:46,115 DEBUG [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:46,115 INFO [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(1838): Closed testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:46,115 DEBUG [RegionOpenAndInit-testSnapshotsFromNamespaces6-pool-0] regionserver.HRegion(1558): Region close journal for ba20086e056a19e754e00b6bd66152a9: 2023-07-12 19:40:46,118 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=51, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSnapshotsFromNamespaces6 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:46,118 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.","families":{"info":[{"qualifier":"regioninfo","vlen":62,"tag":[],"timestamp":"1689190846118"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190846118"}]},"ts":"1689190846118"} 2023-07-12 19:40:46,120 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:46,120 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=51, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSnapshotsFromNamespaces6 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:46,121 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotsFromNamespaces6","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190846120"}]},"ts":"1689190846120"} 2023-07-12 19:40:46,122 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotsFromNamespaces6, state=ENABLING in hbase:meta 2023-07-12 19:40:46,125 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotsFromNamespaces6, region=ba20086e056a19e754e00b6bd66152a9, ASSIGN}] 2023-07-12 19:40:46,127 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSnapshotsFromNamespaces6, region=ba20086e056a19e754e00b6bd66152a9, ASSIGN 2023-07-12 19:40:46,127 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSnapshotsFromNamespaces6, region=ba20086e056a19e754e00b6bd66152a9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:46,278 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=ba20086e056a19e754e00b6bd66152a9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:46,279 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.","families":{"info":[{"qualifier":"regioninfo","vlen":62,"tag":[],"timestamp":"1689190846278"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190846278"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190846278"}]},"ts":"1689190846278"} 2023-07-12 19:40:46,281 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=53, ppid=52, state=RUNNABLE; OpenRegionProcedure ba20086e056a19e754e00b6bd66152a9, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:46,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 19:40:46,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:46,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ba20086e056a19e754e00b6bd66152a9, NAME => 'testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:46,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSnapshotsFromNamespaces6 ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:46,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:46,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:46,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:46,437 INFO [StoreOpener-ba20086e056a19e754e00b6bd66152a9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:46,439 DEBUG [StoreOpener-ba20086e056a19e754e00b6bd66152a9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9/f1 2023-07-12 19:40:46,439 DEBUG [StoreOpener-ba20086e056a19e754e00b6bd66152a9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9/f1 2023-07-12 19:40:46,439 INFO [StoreOpener-ba20086e056a19e754e00b6bd66152a9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ba20086e056a19e754e00b6bd66152a9 columnFamilyName f1 2023-07-12 19:40:46,439 INFO [StoreOpener-ba20086e056a19e754e00b6bd66152a9-1] regionserver.HStore(310): Store=ba20086e056a19e754e00b6bd66152a9/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:46,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:46,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:46,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:40:46,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:46,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ba20086e056a19e754e00b6bd66152a9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70612730, jitterRate=0.05221167206764221}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:46,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ba20086e056a19e754e00b6bd66152a9: 2023-07-12 19:40:46,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9., pid=53, masterSystemTime=1689190846432 2023-07-12 19:40:46,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:46,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:46,448 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=ba20086e056a19e754e00b6bd66152a9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:46,448 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.","families":{"info":[{"qualifier":"regioninfo","vlen":62,"tag":[],"timestamp":"1689190846448"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190846448"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190846448"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190846448"}]},"ts":"1689190846448"} 2023-07-12 19:40:46,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=53, resume processing ppid=52 2023-07-12 19:40:46,451 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=52, state=SUCCESS; OpenRegionProcedure ba20086e056a19e754e00b6bd66152a9, server=jenkins-hbase4.apache.org,44631,1689190811374 in 168 msec 2023-07-12 19:40:46,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-07-12 19:40:46,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; TransitRegionStateProcedure table=testSnapshotsFromNamespaces6, region=ba20086e056a19e754e00b6bd66152a9, ASSIGN in 326 msec 2023-07-12 19:40:46,452 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=51, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSnapshotsFromNamespaces6 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:46,452 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSnapshotsFromNamespaces6","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190846452"}]},"ts":"1689190846452"} 2023-07-12 19:40:46,454 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSnapshotsFromNamespaces6, state=ENABLED in hbase:meta 2023-07-12 19:40:46,456 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=51, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSnapshotsFromNamespaces6 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:46,457 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, state=SUCCESS; CreateTableProcedure table=testSnapshotsFromNamespaces6 in 1.1650 sec 2023-07-12 19:40:46,910 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x079a2ca0 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:46,913 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50dceda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:46,915 DEBUG [hconnection-0xae6e16-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:46,917 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:47,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=51 2023-07-12 19:40:47,400 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testSnapshotsFromNamespaces6, procId: 51 completed 2023-07-12 19:40:47,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190847419 (current time:1689190847419). 2023-07-12 19:40:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbb4198 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:47,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7cf76577, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:47,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:47,424 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:47,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbb4198 to 127.0.0.1:57973 2023-07-12 19:40:47,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:47,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:47,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:47,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=EXCLUSIVE 2023-07-12 19:40:47,428 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=54, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=EXCLUSIVE 2023-07-12 19:40:47,429 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=54, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=EXCLUSIVE; timeout=600000, timestamp=1689191447429 2023-07-12 19:40:47,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:47,429 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces4snapshot C_M_SNAPSHOT_TABLE on table snapshots_from_namespaces:testSnapshotsFromNamespaces4 2023-07-12 19:40:47,430 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:47,430 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:47,431 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=54, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=EXCLUSIVE 2023-07-12 19:40:47,431 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=55, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=SHARED 2023-07-12 19:40:47,432 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=EXCLUSIVE in 4 msec 2023-07-12 19:40:47,432 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=55, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=SHARED 2023-07-12 19:40:47,433 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=55, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=SHARED; timeout=600000, timestamp=1689191447433 2023-07-12 19:40:47,530 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:47,531 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:47,732 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:47,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:47,733 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:47,841 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,841 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromNamespaces4snapshot' 2023-07-12 19:40:47,841 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:47,841 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromNamespaces4snapshot' starting 'acquire' 2023-07-12 19:40:47,841 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromNamespaces4snapshot', kicking off acquire phase on members. 2023-07-12 19:40:47,841 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,842 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,844 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:47,844 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:47,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:47,844 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:47,844 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 111 2023-07-12 19:40:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,845 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromNamespaces4snapshot from table snapshots_from_namespaces:testSnapshotsFromNamespaces4 type SKIPFLUSH 2023-07-12 19:40:47,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromNamespaces4snapshot' with timeout 300000ms 2023-07-12 19:40:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromNamespaces4snapshot' starting 'acquire' stage 2023-07-12 19:40:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromNamespaces4snapshot' locally acquired 2023-07-12 19:40:47,846 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromNamespaces4snapshot) in zk 2023-07-12 19:40:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,848 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromNamespaces4snapshot' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:47,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromNamespaces4snapshot' on coordinator 2023-07-12 19:40:47,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69eb38ed[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:47,851 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromNamespaces4snapshot' starting 'in-barrier' execution. 2023-07-12 19:40:47,851 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,852 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromNamespaces4snapshot' received 'reached' from coordinator. 2023-07-12 19:40:47,852 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:47,852 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:47,852 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:47,852 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:47,853 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2.' region-info for snapshot=testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:47,853 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:47,853 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:47,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x430d31d4 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:47,912 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a7ec727, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:47,913 DEBUG [hconnection-0x4f5ef5f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:47,917 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:47,948 INFO [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(402): snapshots_from_namespaces:testSnapshotsFromNamespaces5 moving into observance of namespace space quota 2023-07-12 19:40:47,950 INFO [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(402): snapshots_from_namespaces:testSnapshotsFromNamespaces4 moving into observance of namespace space quota 2023-07-12 19:40:48,033 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:48,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:48,034 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:48,264 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. completed. 2023-07-12 19:40:48,265 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:40:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromNamespaces4snapshot' locally completed 2023-07-12 19:40:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromNamespaces4snapshot' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:48,267 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromNamespaces4snapshot' has notified controller of completion 2023-07-12 19:40:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:48,267 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromNamespaces4snapshot' completed. 2023-07-12 19:40:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromNamespaces4snapshot' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:48,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromNamespaces4snapshot', counting down latch. Waiting for 0 more 2023-07-12 19:40:48,270 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromNamespaces4snapshot' execution completed 2023-07-12 19:40:48,270 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:48,270 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:48,270 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,270 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromNamespaces4snapshotincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:48,272 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,272 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,272 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:48,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:48,272 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,274 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,276 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:48,277 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:48,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:48,277 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:48,277 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:48,277 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:48,278 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,282 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,282 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,282 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,282 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:48,282 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,282 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,282 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,535 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:48,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:48,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:48,536 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:48,696 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces4snapshot to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:48,923 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSnapshotsFromNamespaces6' 2023-07-12 19:40:48,924 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'snapshots_from_namespaces:testSnapshotsFromNamespaces5' 2023-07-12 19:40:48,924 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'snapshots_from_namespaces:testSnapshotsFromNamespaces4' 2023-07-12 19:40:48,991 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1a67f4d8 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:48,994 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3969dfcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:48,995 DEBUG [hconnection-0x6980bce6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:48,996 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:49,034 DEBUG [master/jenkins-hbase4:0.Chore.1] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:49,034 ERROR [master/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(166): Caught error java.lang.NullPointerException at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.getStoreFilesFromSnapshot(FileArchiverNotifierImpl.java:488) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeSnapshotSizes(FileArchiverNotifierImpl.java:404) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeAndStoreSnapshotSizes(FileArchiverNotifierImpl.java:334) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.computeSnapshotSizes(SnapshotQuotaObserverChore.java:225) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:118) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:49,436 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 19:40:49,513 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromNamespaces4snapshot of table snapshots_from_namespaces:testSnapshotsFromNamespaces4 completed 2023-07-12 19:40:49,513 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:49,513 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces4snapshot 2023-07-12 19:40:49,513 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces4snapshot C_M_SNAPSHOT_TABLE on table snapshots_from_namespaces:testSnapshotsFromNamespaces4 at 1689190847429Consolidate snapshot: testSnapshotsFromNamespaces4snapshot at 1689190848277 (+848 ms)Loading Region manifests for testSnapshotsFromNamespaces4snapshot at 1689190848277Writing data manifest for testSnapshotsFromNamespaces4snapshot at 1689190848280 (+3 ms)Verifying snapshot: testSnapshotsFromNamespaces4snapshot at 1689190848690 (+410 ms)Snapshot testSnapshotsFromNamespaces4snapshot of table snapshots_from_namespaces:testSnapshotsFromNamespaces4 completed at 1689190849513 (+823 ms) 2023-07-12 19:40:49,514 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=55, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=SHARED 2023-07-12 19:40:49,516 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces4, type=SHARED in 2.0840 sec 2023-07-12 19:40:49,536 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:49,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:49,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromNamespaces4snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces4 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:49,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190849538 (current time:1689190849538). 2023-07-12 19:40:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x34efea28 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:49,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@224ecaac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:49,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:49,545 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:49,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34efea28 to 127.0.0.1:57973 2023-07-12 19:40:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:49,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=56, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE 2023-07-12 19:40:49,549 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=56, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE 2023-07-12 19:40:49,550 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=56, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE; timeout=600000, timestamp=1689191449550 2023-07-12 19:40:49,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:49,550 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces5snapshot C_M_SNAPSHOT_TABLE on table snapshots_from_namespaces:testSnapshotsFromNamespaces5 2023-07-12 19:40:49,552 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:49,552 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:49,552 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED 2023-07-12 19:40:49,553 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=56, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE 2023-07-12 19:40:49,554 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE in 5 msec 2023-07-12 19:40:49,555 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=57, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED 2023-07-12 19:40:49,556 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=57, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED; timeout=600000, timestamp=1689191449556 2023-07-12 19:40:49,564 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,564 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromNamespaces5snapshot' 2023-07-12 19:40:49,564 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:49,564 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromNamespaces5snapshot' starting 'acquire' 2023-07-12 19:40:49,564 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromNamespaces5snapshot', kicking off acquire phase on members. 2023-07-12 19:40:49,565 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,565 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,566 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:49,566 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:49,567 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,567 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 111 2023-07-12 19:40:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,568 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromNamespaces5snapshot from table snapshots_from_namespaces:testSnapshotsFromNamespaces5 type SKIPFLUSH 2023-07-12 19:40:49,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromNamespaces5snapshot' with timeout 300000ms 2023-07-12 19:40:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:49,569 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromNamespaces5snapshot' starting 'acquire' stage 2023-07-12 19:40:49,569 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromNamespaces5snapshot' locally acquired 2023-07-12 19:40:49,569 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromNamespaces5snapshot) in zk 2023-07-12 19:40:49,574 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,574 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromNamespaces5snapshot' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:49,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromNamespaces5snapshot' on coordinator 2023-07-12 19:40:49,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f8dfd3[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:49,589 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromNamespaces5snapshot' starting 'in-barrier' execution. 2023-07-12 19:40:49,589 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,590 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,590 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromNamespaces5snapshot' received 'reached' from coordinator. 2023-07-12 19:40:49,590 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,590 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:49,591 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:49,591 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:49,591 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:49,591 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:49,591 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.' region-info for snapshot=testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,591 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:49,591 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:49,610 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. completed. 2023-07-12 19:40:49,611 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:49,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:49,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:49,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromNamespaces5snapshot' locally completed 2023-07-12 19:40:49,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromNamespaces5snapshot' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:49,613 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,613 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromNamespaces5snapshot' has notified controller of completion 2023-07-12 19:40:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:49,613 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:49,613 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromNamespaces5snapshot' completed. 2023-07-12 19:40:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromNamespaces5snapshot' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:49,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromNamespaces5snapshot', counting down latch. Waiting for 0 more 2023-07-12 19:40:49,617 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromNamespaces5snapshot' execution completed 2023-07-12 19:40:49,617 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:49,617 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:49,617 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,617 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromNamespaces5snapshotincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:49,619 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,619 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:49,619 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:49,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:49,620 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,622 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,627 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,627 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:49,627 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,627 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,627 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,627 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,627 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:49,627 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:49,628 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:49,628 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,628 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:49,628 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:49,628 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:49,652 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:49,653 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:49,853 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:49,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:49,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:49,855 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:49,956 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x51356010 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:49,959 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7acff749, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:49,960 DEBUG [hconnection-0x679dce0b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:49,961 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:50,049 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces5snapshot to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:50,071 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromNamespaces5snapshot of table snapshots_from_namespaces:testSnapshotsFromNamespaces5 completed 2023-07-12 19:40:50,071 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:50,072 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces5snapshot 2023-07-12 19:40:50,072 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces5snapshot C_M_SNAPSHOT_TABLE on table snapshots_from_namespaces:testSnapshotsFromNamespaces5 at 1689190849550Consolidate snapshot: testSnapshotsFromNamespaces5snapshot at 1689190849627 (+77 ms)Loading Region manifests for testSnapshotsFromNamespaces5snapshot at 1689190849628 (+1 ms)Writing data manifest for testSnapshotsFromNamespaces5snapshot at 1689190849631 (+3 ms)Verifying snapshot: testSnapshotsFromNamespaces5snapshot at 1689190850044 (+413 ms)Snapshot testSnapshotsFromNamespaces5snapshot of table snapshots_from_namespaces:testSnapshotsFromNamespaces5 completed at 1689190850071 (+27 ms) 2023-07-12 19:40:50,073 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=57, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED 2023-07-12 19:40:50,075 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED in 522 msec 2023-07-12 19:40:50,155 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromNamespaces5snapshot table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:50,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190850157 (current time:1689190850157). 2023-07-12 19:40:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:50,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x268c2f45 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:50,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6df92f16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:50,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:50,168 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52170, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:50,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x268c2f45 to 127.0.0.1:57973 2023-07-12 19:40:50,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:50,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:50,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:50,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=58, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE 2023-07-12 19:40:50,175 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=58, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE 2023-07-12 19:40:50,175 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=58, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE; timeout=600000, timestamp=1689191450175 2023-07-12 19:40:50,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:50,175 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces6snapshot C_M_SNAPSHOT_TABLE on table testSnapshotsFromNamespaces6 2023-07-12 19:40:50,177 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:50,177 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:50,178 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=58, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE 2023-07-12 19:40:50,178 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=59, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED 2023-07-12 19:40:50,179 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=58, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE in 5 msec 2023-07-12 19:40:50,180 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=59, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED 2023-07-12 19:40:50,180 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=59, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED; timeout=600000, timestamp=1689191450180 2023-07-12 19:40:50,278 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:50,279 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:50,479 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:50,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:50,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:50,481 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:50,588 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,588 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromNamespaces6snapshot' 2023-07-12 19:40:50,589 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:50,589 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromNamespaces6snapshot' starting 'acquire' 2023-07-12 19:40:50,589 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromNamespaces6snapshot', kicking off acquire phase on members. 2023-07-12 19:40:50,589 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,589 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,592 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,592 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:50,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:50,592 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,592 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 85 2023-07-12 19:40:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,593 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromNamespaces6snapshot from table testSnapshotsFromNamespaces6 type SKIPFLUSH 2023-07-12 19:40:50,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromNamespaces6snapshot' with timeout 300000ms 2023-07-12 19:40:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromNamespaces6snapshot' starting 'acquire' stage 2023-07-12 19:40:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromNamespaces6snapshot' locally acquired 2023-07-12 19:40:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromNamespaces6snapshot) in zk 2023-07-12 19:40:50,596 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromNamespaces6snapshot' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromNamespaces6snapshot' on coordinator 2023-07-12 19:40:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6685d2c[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:50,598 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromNamespaces6snapshot' starting 'in-barrier' execution. 2023-07-12 19:40:50,598 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,599 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromNamespaces6snapshot' received 'reached' from coordinator. 2023-07-12 19:40:50,600 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,600 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:50,600 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:50,600 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:50,600 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:50,600 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.' region-info for snapshot=testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,600 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:50,600 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:50,611 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. completed. 2023-07-12 19:40:50,611 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromNamespaces6snapshot' locally completed 2023-07-12 19:40:50,611 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromNamespaces6snapshot' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromNamespaces6snapshot' has notified controller of completion 2023-07-12 19:40:50,613 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromNamespaces6snapshot' completed. 2023-07-12 19:40:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromNamespaces6snapshot' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:50,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromNamespaces6snapshot', counting down latch. Waiting for 0 more 2023-07-12 19:40:50,617 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromNamespaces6snapshot' execution completed 2023-07-12 19:40:50,617 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:50,617 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:50,617 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,617 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromNamespaces6snapshotincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:50,620 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,620 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,620 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:50,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:50,621 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,623 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,625 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,625 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:50,625 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,625 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:50,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:50,625 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,625 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,626 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,625 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:50,626 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:50,626 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:50,627 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,627 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,627 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:50,781 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:50,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:50,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:50,783 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:50,916 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x43e43ce3 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:50,919 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@632286a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:50,927 DEBUG [hconnection-0x497ce6fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:50,929 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53546, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:51,043 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces6snapshot to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:51,283 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:51,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:51,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:51,284 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:51,860 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromNamespaces6snapshot of table testSnapshotsFromNamespaces6 completed 2023-07-12 19:40:51,860 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:51,861 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces6snapshot 2023-07-12 19:40:51,861 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces6snapshot C_M_SNAPSHOT_TABLE on table testSnapshotsFromNamespaces6 at 1689190850176Consolidate snapshot: testSnapshotsFromNamespaces6snapshot at 1689190850626 (+450 ms)Loading Region manifests for testSnapshotsFromNamespaces6snapshot at 1689190850626Writing data manifest for testSnapshotsFromNamespaces6snapshot at 1689190850629 (+3 ms)Verifying snapshot: testSnapshotsFromNamespaces6snapshot at 1689190851037 (+408 ms)Snapshot testSnapshotsFromNamespaces6snapshot of table testSnapshotsFromNamespaces6 completed at 1689190851860 (+823 ms) 2023-07-12 19:40:51,862 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=59, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED 2023-07-12 19:40:51,864 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED in 1.6850 sec 2023-07-12 19:40:51,982 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x770729cc to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:51,999 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ec11fe7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:52,006 DEBUG [hconnection-0x7cfa165f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:52,008 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53550, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:52,284 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:52,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:52,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromNamespaces6snapshot table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:52,286 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x120b1f55 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:52,290 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370d006b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:52,291 DEBUG [hconnection-0x3d46e0c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:52,294 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:52,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:52,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190852325 (current time:1689190852325). 2023-07-12 19:40:52,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:52,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x77a55f90 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:52,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f7e65b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:52,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:52,332 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53578, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:52,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77a55f90 to 127.0.0.1:57973 2023-07-12 19:40:52,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:52,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:52,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:52,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=60, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE 2023-07-12 19:40:52,347 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=60, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE 2023-07-12 19:40:52,348 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=60, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE; timeout=600000, timestamp=1689191452348 2023-07-12 19:40:52,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:52,349 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces5snapshot1 C_M_SNAPSHOT_TABLE on table snapshots_from_namespaces:testSnapshotsFromNamespaces5 2023-07-12 19:40:52,350 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:52,351 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=60, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE 2023-07-12 19:40:52,351 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=61, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED 2023-07-12 19:40:52,352 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=EXCLUSIVE in 10 msec 2023-07-12 19:40:52,351 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:52,354 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=61, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED 2023-07-12 19:40:52,355 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=61, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED; timeout=600000, timestamp=1689191452355 2023-07-12 19:40:52,454 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:52,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:52,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:52,456 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:52,656 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:52,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:52,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:52,658 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:52,784 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,784 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromNamespaces5snapshot1' 2023-07-12 19:40:52,784 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:52,784 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromNamespaces5snapshot1' starting 'acquire' 2023-07-12 19:40:52,784 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromNamespaces5snapshot1', kicking off acquire phase on members. 2023-07-12 19:40:52,784 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,785 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,786 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:52,786 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:52,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:52,786 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:52,786 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 112 2023-07-12 19:40:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,787 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromNamespaces5snapshot1 from table snapshots_from_namespaces:testSnapshotsFromNamespaces5 type SKIPFLUSH 2023-07-12 19:40:52,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromNamespaces5snapshot1' with timeout 300000ms 2023-07-12 19:40:52,787 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:52,788 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromNamespaces5snapshot1' starting 'acquire' stage 2023-07-12 19:40:52,788 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromNamespaces5snapshot1' locally acquired 2023-07-12 19:40:52,788 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromNamespaces5snapshot1) in zk 2023-07-12 19:40:52,790 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,790 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:52,790 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,790 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromNamespaces5snapshot1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:52,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromNamespaces5snapshot1' on coordinator 2023-07-12 19:40:52,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c93bff[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:52,792 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromNamespaces5snapshot1' starting 'in-barrier' execution. 2023-07-12 19:40:52,792 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,793 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,793 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromNamespaces5snapshot1' received 'reached' from coordinator. 2023-07-12 19:40:52,793 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:52,793 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:52,794 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:52,794 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:52,794 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:52,794 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:52,794 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f.' region-info for snapshot=testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:52,794 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:52,794 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:52,958 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:52,958 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5515791f to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:52,959 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:52,961 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60e284f7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:52,963 DEBUG [hconnection-0x6d790346-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:52,964 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53594, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:53,199 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. completed. 2023-07-12 19:40:53,200 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:40:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromNamespaces5snapshot1' locally completed 2023-07-12 19:40:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromNamespaces5snapshot1' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:53,201 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,201 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromNamespaces5snapshot1' has notified controller of completion 2023-07-12 19:40:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:53,201 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:53,202 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromNamespaces5snapshot1' completed. 2023-07-12 19:40:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromNamespaces5snapshot1' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:53,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromNamespaces5snapshot1', counting down latch. Waiting for 0 more 2023-07-12 19:40:53,205 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromNamespaces5snapshot1' execution completed 2023-07-12 19:40:53,205 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:53,205 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:53,205 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,205 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromNamespaces5snapshot1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:53,210 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,210 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:53,210 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:53,211 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,212 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,215 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,215 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,215 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:53,215 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:53,215 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:53,215 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 because node does not exist (not an error) 2023-07-12 19:40:53,215 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:53,215 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:53,215 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:53,215 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,216 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,216 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:53,217 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,459 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:53,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:53,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:53,461 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:53,630 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces5snapshot1 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,908 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x37698ef5 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:53,911 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cf6df6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:53,913 DEBUG [hconnection-0x3b57b555-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:53,914 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53608, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:53,955 DEBUG [master/jenkins-hbase4:0.Chore.1] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:53,955 ERROR [master/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(166): Caught error java.lang.NullPointerException at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.getStoreFilesFromSnapshot(FileArchiverNotifierImpl.java:488) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeSnapshotSizes(FileArchiverNotifierImpl.java:404) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeAndStoreSnapshotSizes(FileArchiverNotifierImpl.java:334) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.computeSnapshotSizes(SnapshotQuotaObserverChore.java:225) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:118) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:40:54,446 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromNamespaces5snapshot1 of table snapshots_from_namespaces:testSnapshotsFromNamespaces5 completed 2023-07-12 19:40:54,447 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:54,447 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces5snapshot1 2023-07-12 19:40:54,447 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces5snapshot1 C_M_SNAPSHOT_TABLE on table snapshots_from_namespaces:testSnapshotsFromNamespaces5 at 1689190852349Consolidate snapshot: testSnapshotsFromNamespaces5snapshot1 at 1689190853215 (+866 ms)Loading Region manifests for testSnapshotsFromNamespaces5snapshot1 at 1689190853215Writing data manifest for testSnapshotsFromNamespaces5snapshot1 at 1689190853219 (+4 ms)Verifying snapshot: testSnapshotsFromNamespaces5snapshot1 at 1689190853625 (+406 ms)Snapshot testSnapshotsFromNamespaces5snapshot1 of table snapshots_from_namespaces:testSnapshotsFromNamespaces5 completed at 1689190854446 (+821 ms) 2023-07-12 19:40:54,449 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=61, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED 2023-07-12 19:40:54,450 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=snapshots_from_namespaces:testSnapshotsFromNamespaces5, type=SHARED in 2.0990 sec 2023-07-12 19:40:54,461 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:54,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:54,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromNamespaces5snapshot1 table=snapshots_from_namespaces:testSnapshotsFromNamespaces5 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:54,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:54,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190854463 (current time:1689190854463). 2023-07-12 19:40:54,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:54,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x79448901 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:54,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24400066, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:54,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:54,469 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:54,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79448901 to 127.0.0.1:57973 2023-07-12 19:40:54,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE 2023-07-12 19:40:54,473 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE 2023-07-12 19:40:54,473 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE; timeout=600000, timestamp=1689191454473 2023-07-12 19:40:54,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:54,473 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces6snapshot2 C_M_SNAPSHOT_TABLE on table testSnapshotsFromNamespaces6 2023-07-12 19:40:54,474 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:54,474 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:54,475 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE 2023-07-12 19:40:54,475 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED 2023-07-12 19:40:54,476 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=EXCLUSIVE in 4 msec 2023-07-12 19:40:54,476 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=63, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED 2023-07-12 19:40:54,477 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=63, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED; timeout=600000, timestamp=1689191454477 2023-07-12 19:40:54,574 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:54,575 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:54,776 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:54,777 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:54,884 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,884 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSnapshotsFromNamespaces6snapshot2' 2023-07-12 19:40:54,884 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:54,884 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSnapshotsFromNamespaces6snapshot2' starting 'acquire' 2023-07-12 19:40:54,884 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSnapshotsFromNamespaces6snapshot2', kicking off acquire phase on members. 2023-07-12 19:40:54,885 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,885 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,886 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:54,886 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:54,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:54,886 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:54,886 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 86 2023-07-12 19:40:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,887 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testSnapshotsFromNamespaces6snapshot2 from table testSnapshotsFromNamespaces6 type SKIPFLUSH 2023-07-12 19:40:54,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,887 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSnapshotsFromNamespaces6snapshot2' with timeout 300000ms 2023-07-12 19:40:54,888 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:54,888 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSnapshotsFromNamespaces6snapshot2' starting 'acquire' stage 2023-07-12 19:40:54,888 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSnapshotsFromNamespaces6snapshot2' locally acquired 2023-07-12 19:40:54,888 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testSnapshotsFromNamespaces6snapshot2) in zk 2023-07-12 19:40:54,890 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:54,890 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSnapshotsFromNamespaces6snapshot2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:54,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testSnapshotsFromNamespaces6snapshot2' on coordinator 2023-07-12 19:40:54,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71554f7b[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:54,892 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSnapshotsFromNamespaces6snapshot2' starting 'in-barrier' execution. 2023-07-12 19:40:54,892 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,893 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSnapshotsFromNamespaces6snapshot2' received 'reached' from coordinator. 2023-07-12 19:40:54,893 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:54,893 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:54,893 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:54,894 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:54,894 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:54,894 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9.' region-info for snapshot=testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:54,894 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:54,894 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 19:40:54,921 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2acb1b19 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:54,924 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a391d83, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:54,925 DEBUG [hconnection-0x7f8e177-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:54,927 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53618, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:55,077 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:55,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:55,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:55,078 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:55,300 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. completed. 2023-07-12 19:40:55,300 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:40:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSnapshotsFromNamespaces6snapshot2' locally completed 2023-07-12 19:40:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSnapshotsFromNamespaces6snapshot2' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:55,302 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSnapshotsFromNamespaces6snapshot2' has notified controller of completion 2023-07-12 19:40:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSnapshotsFromNamespaces6snapshot2' completed. 2023-07-12 19:40:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSnapshotsFromNamespaces6snapshot2' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:55,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testSnapshotsFromNamespaces6snapshot2', counting down latch. Waiting for 0 more 2023-07-12 19:40:55,307 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSnapshotsFromNamespaces6snapshot2' execution completed 2023-07-12 19:40:55,307 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:55,307 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:55,307 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,308 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSnapshotsFromNamespaces6snapshot2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:55,311 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,311 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:55,311 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:55,313 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,316 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,318 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,318 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,318 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:55,318 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:55,318 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:55,318 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,318 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,318 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:55,321 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:55,322 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,322 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,322 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:55,322 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,346 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces6snapshot2 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:55,579 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:55,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:55,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:55,580 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:40:55,915 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x792476e5 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:55,918 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4175713f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:55,920 DEBUG [hconnection-0x41464aea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:55,921 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:56,168 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testSnapshotsFromNamespaces6snapshot2 of table testSnapshotsFromNamespaces6 completed 2023-07-12 19:40:56,168 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:56,168 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testSnapshotsFromNamespaces6snapshot2 2023-07-12 19:40:56,168 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testSnapshotsFromNamespaces6snapshot2 C_M_SNAPSHOT_TABLE on table testSnapshotsFromNamespaces6 at 1689190854473Consolidate snapshot: testSnapshotsFromNamespaces6snapshot2 at 1689190855318 (+845 ms)Loading Region manifests for testSnapshotsFromNamespaces6snapshot2 at 1689190855322 (+4 ms)Writing data manifest for testSnapshotsFromNamespaces6snapshot2 at 1689190855326 (+4 ms)Verifying snapshot: testSnapshotsFromNamespaces6snapshot2 at 1689190855340 (+14 ms)Snapshot testSnapshotsFromNamespaces6snapshot2 of table testSnapshotsFromNamespaces6 completed at 1689190856168 (+828 ms) 2023-07-12 19:40:56,169 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=63, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED 2023-07-12 19:40:56,171 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testSnapshotsFromNamespaces6, type=SHARED in 1.6950 sec 2023-07-12 19:40:56,580 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:56,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:56,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testSnapshotsFromNamespaces6snapshot2 table=testSnapshotsFromNamespaces6 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:40:56,582 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x01a78a98 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:56,585 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f3f7128, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:56,586 DEBUG [hconnection-0x54784ef0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:56,589 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:56,626 INFO [Listener at localhost/32953] hbase.ResourceChecker(175): after: quotas.TestSnapshotQuotaObserverChore#testSnapshotsFromNamespaces Thread=296 (was 310), OpenFileDescriptor=613 (was 617), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=122 (was 96) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=7986 (was 8142) 2023-07-12 19:40:56,639 INFO [Listener at localhost/32953] hbase.ResourceChecker(147): before: quotas.TestSnapshotQuotaObserverChore#testPersistingSnapshotsForNamespaces Thread=296, OpenFileDescriptor=613, MaxFileDescriptor=60000, SystemLoadAverage=122, ProcessCount=172, AvailableMemoryMB=7986 2023-07-12 19:40:56,641 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x1aa8ec38 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:56,645 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16706280, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:56,646 DEBUG [hconnection-0x302a6c1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:56,648 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:56,651 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(444): Deleting quota for namespace: default 2023-07-12 19:40:56,653 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(444): Deleting quota for namespace: snapshots_from_namespaces 2023-07-12 19:40:56,659 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(450): Deleting quota for user: user 2023-07-12 19:40:56,660 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aa8ec38 to 127.0.0.1:57973 2023-07-12 19:40:56,660 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:56,673 INFO [Listener at localhost/32953] hbase.ResourceChecker(175): after: quotas.TestSnapshotQuotaObserverChore#testPersistingSnapshotsForNamespaces Thread=301 (was 296) - Thread LEAK? -, OpenFileDescriptor=618 (was 613) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=122 (was 122), ProcessCount=172 (was 172), AvailableMemoryMB=7985 (was 7986) 2023-07-12 19:40:56,685 INFO [Listener at localhost/32953] hbase.ResourceChecker(147): before: quotas.TestSnapshotQuotaObserverChore#testBucketingFilesToSnapshots Thread=301, OpenFileDescriptor=617, MaxFileDescriptor=60000, SystemLoadAverage=122, ProcessCount=172, AvailableMemoryMB=7985 2023-07-12 19:40:56,687 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x0576f7f6 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:56,690 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1144007c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:56,691 DEBUG [hconnection-0x597fb9cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:56,693 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:56,696 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0576f7f6 to 127.0.0.1:57973 2023-07-12 19:40:56,696 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:56,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testBucketingFilesToSnapshots7', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:40:56,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=64, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testBucketingFilesToSnapshots7 2023-07-12 19:40:56,700 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=64, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testBucketingFilesToSnapshots7 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:40:56,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testBucketingFilesToSnapshots7" procId is: 64 2023-07-12 19:40:56,701 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=64, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testBucketingFilesToSnapshots7 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:40:56,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=64 2023-07-12 19:40:56,702 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:56,703 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa empty. 2023-07-12 19:40:56,704 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:56,704 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testBucketingFilesToSnapshots7 regions 2023-07-12 19:40:56,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=64 2023-07-12 19:40:56,908 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x245344ed to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:56,911 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28b77949, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:56,912 DEBUG [hconnection-0x7201e1e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:56,913 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:57,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=64 2023-07-12 19:40:57,123 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testBucketingFilesToSnapshots7/.tabledesc/.tableinfo.0000000001 2023-07-12 19:40:57,124 INFO [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(7675): creating {ENCODED => af83dc5342faa5fe20a6e4dc7a578aaa, NAME => 'testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testBucketingFilesToSnapshots7', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:40:57,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=64 2023-07-12 19:40:57,530 DEBUG [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(866): Instantiated testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:57,530 DEBUG [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(1604): Closing af83dc5342faa5fe20a6e4dc7a578aaa, disabling compactions & flushes 2023-07-12 19:40:57,530 INFO [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(1626): Closing region testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:57,531 DEBUG [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:57,531 DEBUG [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(1714): Acquired close lock on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. after waiting 0 ms 2023-07-12 19:40:57,531 DEBUG [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(1724): Updates disabled for region testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:57,531 INFO [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(1838): Closed testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:57,531 DEBUG [RegionOpenAndInit-testBucketingFilesToSnapshots7-pool-0] regionserver.HRegion(1558): Region close journal for af83dc5342faa5fe20a6e4dc7a578aaa: 2023-07-12 19:40:57,533 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=64, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testBucketingFilesToSnapshots7 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:40:57,535 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689190857534"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190857534"}]},"ts":"1689190857534"} 2023-07-12 19:40:57,536 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:40:57,537 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=64, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testBucketingFilesToSnapshots7 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:40:57,537 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testBucketingFilesToSnapshots7","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190857537"}]},"ts":"1689190857537"} 2023-07-12 19:40:57,538 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testBucketingFilesToSnapshots7, state=ENABLING in hbase:meta 2023-07-12 19:40:57,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=65, ppid=64, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testBucketingFilesToSnapshots7, region=af83dc5342faa5fe20a6e4dc7a578aaa, ASSIGN}] 2023-07-12 19:40:57,545 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=64, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testBucketingFilesToSnapshots7, region=af83dc5342faa5fe20a6e4dc7a578aaa, ASSIGN 2023-07-12 19:40:57,545 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=64, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testBucketingFilesToSnapshots7, region=af83dc5342faa5fe20a6e4dc7a578aaa, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:40:57,696 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=af83dc5342faa5fe20a6e4dc7a578aaa, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:57,697 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689190857696"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190857696"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190857696"}]},"ts":"1689190857696"} 2023-07-12 19:40:57,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=66, ppid=65, state=RUNNABLE; OpenRegionProcedure af83dc5342faa5fe20a6e4dc7a578aaa, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:40:57,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=64 2023-07-12 19:40:57,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:57,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => af83dc5342faa5fe20a6e4dc7a578aaa, NAME => 'testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:40:57,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testBucketingFilesToSnapshots7 af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:57,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:40:57,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:57,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:57,854 INFO [StoreOpener-af83dc5342faa5fe20a6e4dc7a578aaa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:57,855 DEBUG [StoreOpener-af83dc5342faa5fe20a6e4dc7a578aaa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1 2023-07-12 19:40:57,855 DEBUG [StoreOpener-af83dc5342faa5fe20a6e4dc7a578aaa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1 2023-07-12 19:40:57,856 INFO [StoreOpener-af83dc5342faa5fe20a6e4dc7a578aaa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region af83dc5342faa5fe20a6e4dc7a578aaa columnFamilyName f1 2023-07-12 19:40:57,856 INFO [StoreOpener-af83dc5342faa5fe20a6e4dc7a578aaa-1] regionserver.HStore(310): Store=af83dc5342faa5fe20a6e4dc7a578aaa/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:40:57,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:57,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:57,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:40:57,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:40:57,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened af83dc5342faa5fe20a6e4dc7a578aaa; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67496273, jitterRate=0.005772843956947327}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:40:57,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for af83dc5342faa5fe20a6e4dc7a578aaa: 2023-07-12 19:40:57,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa., pid=66, masterSystemTime=1689190857850 2023-07-12 19:40:57,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:57,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:57,865 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=af83dc5342faa5fe20a6e4dc7a578aaa, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:57,865 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689190857864"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190857864"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190857864"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190857864"}]},"ts":"1689190857864"} 2023-07-12 19:40:57,867 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=66, resume processing ppid=65 2023-07-12 19:40:57,867 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=65, state=SUCCESS; OpenRegionProcedure af83dc5342faa5fe20a6e4dc7a578aaa, server=jenkins-hbase4.apache.org,44631,1689190811374 in 168 msec 2023-07-12 19:40:57,869 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=65, resume processing ppid=64 2023-07-12 19:40:57,869 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=64, state=SUCCESS; TransitRegionStateProcedure table=testBucketingFilesToSnapshots7, region=af83dc5342faa5fe20a6e4dc7a578aaa, ASSIGN in 324 msec 2023-07-12 19:40:57,869 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=64, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testBucketingFilesToSnapshots7 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:40:57,869 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testBucketingFilesToSnapshots7","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190857869"}]},"ts":"1689190857869"} 2023-07-12 19:40:57,870 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testBucketingFilesToSnapshots7, state=ENABLED in hbase:meta 2023-07-12 19:40:57,872 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=64, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testBucketingFilesToSnapshots7 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:40:57,873 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, state=SUCCESS; CreateTableProcedure table=testBucketingFilesToSnapshots7 in 1.1740 sec 2023-07-12 19:40:57,912 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x29b5faf3 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:57,916 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70556449, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:57,918 DEBUG [hconnection-0x787d458f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:57,919 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:58,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=64 2023-07-12 19:40:58,808 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testBucketingFilesToSnapshots7, procId: 64 completed 2023-07-12 19:40:58,896 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 19:40:58,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(936): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-07-12 19:40:58,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(165): Submitting procedure testBucketingFilesToSnapshots7 2023-07-12 19:40:58,901 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testBucketingFilesToSnapshots7' 2023-07-12 19:40:58,901 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:58,902 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testBucketingFilesToSnapshots7' starting 'acquire' 2023-07-12 19:40:58,902 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testBucketingFilesToSnapshots7', kicking off acquire phase on members. 2023-07-12 19:40:58,903 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,903 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,905 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:58,905 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:58,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:58,906 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,906 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:58,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 19:40:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testBucketingFilesToSnapshots7 2023-07-12 19:40:58,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testBucketingFilesToSnapshots7 2023-07-12 19:40:58,908 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testBucketingFilesToSnapshots7' with timeout 60000ms 2023-07-12 19:40:58,908 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testBucketingFilesToSnapshots7' starting 'acquire' stage 2023-07-12 19:40:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 19:40:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 19:40:58,909 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:58,909 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. started... 2023-07-12 19:40:58,910 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af83dc5342faa5fe20a6e4dc7a578aaa 1/1 column families, dataSize=282.92 KB heapSize=360.25 KB 2023-07-12 19:40:58,950 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=282.92 KB at sequenceid=25 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/.tmp/f1/5ee47a509c6444c78c697978f661c8fa 2023-07-12 19:40:58,957 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/.tmp/f1/5ee47a509c6444c78c697978f661c8fa as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/5ee47a509c6444c78c697978f661c8fa 2023-07-12 19:40:58,963 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/5ee47a509c6444c78c697978f661c8fa, entries=1024, sequenceid=25, filesize=290.9 K 2023-07-12 19:40:58,963 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x419b350a to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:58,964 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~282.92 KB/289706, heapSize ~360.23 KB/368880, currentSize=0 B/0 for af83dc5342faa5fe20a6e4dc7a578aaa in 54ms, sequenceid=25, compaction requested=false 2023-07-12 19:40:58,964 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testBucketingFilesToSnapshots7' 2023-07-12 19:40:58,964 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af83dc5342faa5fe20a6e4dc7a578aaa: 2023-07-12 19:40:58,964 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:58,964 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 19:40:58,965 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 19:40:58,965 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,966 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testBucketingFilesToSnapshots7' locally acquired 2023-07-12 19:40:58,967 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testBucketingFilesToSnapshots7) in zk 2023-07-12 19:40:58,968 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,968 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:58,969 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,969 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testBucketingFilesToSnapshots7' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:58,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testBucketingFilesToSnapshots7' on coordinator 2023-07-12 19:40:58,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7edc10c5[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:58,971 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testBucketingFilesToSnapshots7' starting 'in-barrier' execution. 2023-07-12 19:40:58,971 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,972 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19bfb9d7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:58,972 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,972 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,972 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,972 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testBucketingFilesToSnapshots7' received 'reached' from coordinator. 2023-07-12 19:40:58,972 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testBucketingFilesToSnapshots7' locally completed 2023-07-12 19:40:58,972 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testBucketingFilesToSnapshots7' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:58,975 DEBUG [hconnection-0x60334695-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:58,976 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:58,980 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testBucketingFilesToSnapshots7' has notified controller of completion 2023-07-12 19:40:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testBucketingFilesToSnapshots7' completed. 2023-07-12 19:40:58,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:58,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:58,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:58,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:58,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:58,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testBucketingFilesToSnapshots7' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:58,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testBucketingFilesToSnapshots7', counting down latch. Waiting for 0 more 2023-07-12 19:40:58,985 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testBucketingFilesToSnapshots7' execution completed 2023-07-12 19:40:58,985 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:58,985 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:58,985 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testBucketingFilesToSnapshots7 2023-07-12 19:40:58,985 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testBucketingFilesToSnapshots7including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 19:40:58,986 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,986 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:58,987 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:58,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:58,987 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:58,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:58,990 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,997 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,997 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:58,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testBucketingFilesToSnapshots7' 2023-07-12 19:40:58,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:58,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 19:40:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:58,997 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:58,997 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:58,997 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testBucketingFilesToSnapshots7'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:58,997 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,998 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 19:40:58,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:58,998 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:58,998 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:58,998 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,998 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:58,998 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,014 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testBucketingFilesToSnapshots7 moved into observance of table space quota. 2023-07-12 19:40:59,098 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 19:40:59,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 19:40:59,100 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 19:40:59,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(936): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-07-12 19:40:59,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(143): Procedure testBucketingFilesToSnapshots7 was in running list but was completed. Accepting new attempt. 2023-07-12 19:40:59,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(165): Submitting procedure testBucketingFilesToSnapshots7 2023-07-12 19:40:59,108 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testBucketingFilesToSnapshots7' 2023-07-12 19:40:59,108 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:59,109 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testBucketingFilesToSnapshots7' starting 'acquire' 2023-07-12 19:40:59,109 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testBucketingFilesToSnapshots7', kicking off acquire phase on members. 2023-07-12 19:40:59,110 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,110 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,112 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,112 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:59,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:59,112 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,113 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 19:40:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testBucketingFilesToSnapshots7 2023-07-12 19:40:59,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testBucketingFilesToSnapshots7 is still present, removing 2023-07-12 19:40:59,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testBucketingFilesToSnapshots7 2023-07-12 19:40:59,114 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testBucketingFilesToSnapshots7' with timeout 60000ms 2023-07-12 19:40:59,114 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:40:59,115 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testBucketingFilesToSnapshots7' starting 'acquire' stage 2023-07-12 19:40:59,115 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 19:40:59,115 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 19:40:59,115 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:59,115 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. started... 2023-07-12 19:40:59,116 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af83dc5342faa5fe20a6e4dc7a578aaa: 2023-07-12 19:40:59,116 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 19:40:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 19:40:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testBucketingFilesToSnapshots7' locally acquired 2023-07-12 19:40:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testBucketingFilesToSnapshots7) in zk 2023-07-12 19:40:59,117 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,117 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testBucketingFilesToSnapshots7' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:59,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testBucketingFilesToSnapshots7' on coordinator 2023-07-12 19:40:59,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a99a30f[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:59,120 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testBucketingFilesToSnapshots7' starting 'in-barrier' execution. 2023-07-12 19:40:59,120 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,121 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testBucketingFilesToSnapshots7' received 'reached' from coordinator. 2023-07-12 19:40:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testBucketingFilesToSnapshots7' locally completed 2023-07-12 19:40:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testBucketingFilesToSnapshots7' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:59,121 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,121 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testBucketingFilesToSnapshots7' has notified controller of completion 2023-07-12 19:40:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:59,123 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testBucketingFilesToSnapshots7' completed. 2023-07-12 19:40:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testBucketingFilesToSnapshots7' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:59,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testBucketingFilesToSnapshots7', counting down latch. Waiting for 0 more 2023-07-12 19:40:59,127 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testBucketingFilesToSnapshots7' execution completed 2023-07-12 19:40:59,127 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:59,127 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:59,127 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testBucketingFilesToSnapshots7 2023-07-12 19:40:59,127 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testBucketingFilesToSnapshots7including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 19:40:59,130 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,130 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:40:59,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,130 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:59,130 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:59,132 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7 2023-07-12 19:40:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,135 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:40:59,135 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:40:59,135 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,135 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:40:59,135 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,135 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:59,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 because node does not exist (not an error) 2023-07-12 19:40:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:40:59,135 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testBucketingFilesToSnapshots7' 2023-07-12 19:40:59,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 19:40:59,136 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:40:59,137 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:40:59,137 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBucketingFilesToSnapshots7 2023-07-12 19:40:59,137 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testBucketingFilesToSnapshots7'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:59,137 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 19:40:59,237 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 19:40:59,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 19:40:59,240 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:40:59,244 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testBucketingFilesToSnapshots7: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=290.95 KB/1 GB] 2023-07-12 19:40:59,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:59,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190859245 (current time:1689190859245). 2023-07-12 19:40:59,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:40:59,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x316252f4 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:59,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a729833, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:59,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:59,259 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53722, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:40:59,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x316252f4 to 127.0.0.1:57973 2023-07-12 19:40:59,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:40:59,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:40:59,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:59,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=67, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE 2023-07-12 19:40:59,267 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=67, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE 2023-07-12 19:40:59,268 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=67, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE; timeout=600000, timestamp=1689191459268 2023-07-12 19:40:59,268 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testBucketingFilesToSnapshots7snapshot1 C_M_SNAPSHOT_TABLE on table testBucketingFilesToSnapshots7 2023-07-12 19:40:59,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } 2023-07-12 19:40:59,269 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:40:59,269 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:40:59,270 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=68, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED 2023-07-12 19:40:59,271 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=67, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE 2023-07-12 19:40:59,272 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=67, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE in 5 msec 2023-07-12 19:40:59,273 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=68, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED 2023-07-12 19:40:59,274 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=68, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED; timeout=600000, timestamp=1689191459273 2023-07-12 19:40:59,286 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,288 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testBucketingFilesToSnapshots7snapshot1' 2023-07-12 19:40:59,288 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:59,289 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testBucketingFilesToSnapshots7snapshot1' starting 'acquire' 2023-07-12 19:40:59,289 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testBucketingFilesToSnapshots7snapshot1', kicking off acquire phase on members. 2023-07-12 19:40:59,289 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,290 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,291 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,291 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:59,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:59,291 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,291 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:40:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 90 2023-07-12 19:40:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,293 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testBucketingFilesToSnapshots7snapshot1 from table testBucketingFilesToSnapshots7 type SKIPFLUSH 2023-07-12 19:40:59,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testBucketingFilesToSnapshots7snapshot1' with timeout 300000ms 2023-07-12 19:40:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:40:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testBucketingFilesToSnapshots7snapshot1' starting 'acquire' stage 2023-07-12 19:40:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testBucketingFilesToSnapshots7snapshot1' locally acquired 2023-07-12 19:40:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testBucketingFilesToSnapshots7snapshot1) in zk 2023-07-12 19:40:59,296 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testBucketingFilesToSnapshots7snapshot1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:40:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:59,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testBucketingFilesToSnapshots7snapshot1' on coordinator 2023-07-12 19:40:59,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32876160[Count = 0] remaining members to acquire global barrier 2023-07-12 19:40:59,298 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testBucketingFilesToSnapshots7snapshot1' starting 'in-barrier' execution. 2023-07-12 19:40:59,298 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,306 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,306 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testBucketingFilesToSnapshots7snapshot1' received 'reached' from coordinator. 2023-07-12 19:40:59,306 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:40:59,306 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:40:59,306 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:59,306 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,307 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:40:59,306 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:40:59,307 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.' region-info for snapshot=testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,307 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:40:59,308 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/5ee47a509c6444c78c697978f661c8fa] hfiles 2023-07-12 19:40:59,308 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/5ee47a509c6444c78c697978f661c8fa for snapshot=testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,327 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. completed. 2023-07-12 19:40:59,327 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:40:59,327 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:40:59,328 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:40:59,328 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,328 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testBucketingFilesToSnapshots7snapshot1' locally completed 2023-07-12 19:40:59,328 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testBucketingFilesToSnapshots7snapshot1' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:40:59,330 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testBucketingFilesToSnapshots7snapshot1' has notified controller of completion 2023-07-12 19:40:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testBucketingFilesToSnapshots7snapshot1' completed. 2023-07-12 19:40:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testBucketingFilesToSnapshots7snapshot1' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:40:59,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testBucketingFilesToSnapshots7snapshot1', counting down latch. Waiting for 0 more 2023-07-12 19:40:59,334 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testBucketingFilesToSnapshots7snapshot1' execution completed 2023-07-12 19:40:59,335 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:40:59,335 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:40:59,335 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,335 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testBucketingFilesToSnapshots7snapshot1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:40:59,336 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,336 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:40:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:40:59,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,336 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:40:59,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:59,337 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:40:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,338 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:40:59,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,341 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:40:59,341 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:40:59,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:40:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:40:59,341 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,342 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:40:59,342 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,343 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:40:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:40:59,345 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,342 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,346 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,347 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:40:59,347 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,347 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,347 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,369 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:59,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:59,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:59,371 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:40:59,571 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:59,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:59,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:59,572 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:40:59,772 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testBucketingFilesToSnapshots7snapshot1 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:40:59,872 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:40:59,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:40:59,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:40:59,874 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:40:59,941 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3b5a74cf to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:40:59,944 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@609c6a3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:40:59,946 DEBUG [hconnection-0x37e20588-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:40:59,947 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:00,195 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testBucketingFilesToSnapshots7snapshot1 of table testBucketingFilesToSnapshots7 completed 2023-07-12 19:41:00,195 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:41:00,195 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testBucketingFilesToSnapshots7snapshot1 2023-07-12 19:41:00,195 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testBucketingFilesToSnapshots7snapshot1 C_M_SNAPSHOT_TABLE on table testBucketingFilesToSnapshots7 at 1689190859268Consolidate snapshot: testBucketingFilesToSnapshots7snapshot1 at 1689190859343 (+75 ms)Loading Region manifests for testBucketingFilesToSnapshots7snapshot1 at 1689190859344 (+1 ms)Writing data manifest for testBucketingFilesToSnapshots7snapshot1 at 1689190859349 (+5 ms)Verifying snapshot: testBucketingFilesToSnapshots7snapshot1 at 1689190859766 (+417 ms)Snapshot testBucketingFilesToSnapshots7snapshot1 of table testBucketingFilesToSnapshots7 completed at 1689190860195 (+429 ms) 2023-07-12 19:41:00,197 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=68, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED 2023-07-12 19:41:00,198 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED in 928 msec 2023-07-12 19:41:00,374 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:00,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:00,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testBucketingFilesToSnapshots7snapshot1 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:41:00,375 DEBUG [Listener at localhost/32953] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-07-12 19:41:00,375 DEBUG [Listener at localhost/32953] regionserver.HStore(1912): af83dc5342faa5fe20a6e4dc7a578aaa/f1 is initiating major compaction (all files) 2023-07-12 19:41:00,375 INFO [Listener at localhost/32953] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:41:00,376 INFO [Listener at localhost/32953] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:41:00,376 INFO [Listener at localhost/32953] regionserver.HRegion(2259): Starting compaction of af83dc5342faa5fe20a6e4dc7a578aaa/f1 in testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:00,376 INFO [Listener at localhost/32953] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/5ee47a509c6444c78c697978f661c8fa] into tmpdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/.tmp, totalSize=290.9 K 2023-07-12 19:41:00,377 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting 5ee47a509c6444c78c697978f661c8fa, keycount=1024, bloomtype=ROW, size=290.9 K, encoding=NONE, compression=NONE, seqNum=25, earliestPutTs=1689190858815 2023-07-12 19:41:00,391 INFO [Listener at localhost/32953] throttle.PressureAwareThroughputController(145): af83dc5342faa5fe20a6e4dc7a578aaa#f1#compaction#13 average throughput is 30.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 19:41:00,800 DEBUG [Listener at localhost/32953] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/.tmp/f1/aaf3f083f9744e24a155f9acefc9dc7b as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/aaf3f083f9744e24a155f9acefc9dc7b 2023-07-12 19:41:00,806 INFO [Listener at localhost/32953] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in af83dc5342faa5fe20a6e4dc7a578aaa/f1 of af83dc5342faa5fe20a6e4dc7a578aaa into aaf3f083f9744e24a155f9acefc9dc7b(size=291.0 K), total size for store is 291.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 19:41:00,806 DEBUG [Listener at localhost/32953] regionserver.HRegion(2289): Compaction status journal for af83dc5342faa5fe20a6e4dc7a578aaa: 2023-07-12 19:41:00,807 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:41:00,807 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:00,807 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:00,811 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:00,913 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:00,913 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:00,917 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:00,943 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7245998f to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:00,947 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b734485, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:00,949 DEBUG [hconnection-0x1b9100ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:00,949 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:01,019 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,019 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,023 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,124 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,124 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,128 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,230 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,230 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,235 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,336 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,336 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,341 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,442 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,442 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,447 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,548 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,549 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,553 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,654 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,655 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,659 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,760 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,760 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,764 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,866 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,866 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,871 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,908 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x0a604828 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:01,911 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e11ad41, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:01,912 DEBUG [hconnection-0x121b415d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:01,913 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36868, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:01,972 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:01,972 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:01,977 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:01,977 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testBucketingFilesToSnapshots7 moved into observance of table space quota. 2023-07-12 19:41:02,078 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,078 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,084 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,185 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,186 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,191 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,293 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,293 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,297 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,399 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,399 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,403 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,505 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,505 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,511 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,612 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,613 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,619 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,720 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,720 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,725 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,826 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,827 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,831 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,932 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:02,933 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:02,937 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:02,993 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4399f2e9 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:02,997 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a9eae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:02,998 DEBUG [hconnection-0x3fd6654a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:02,999 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:03,039 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,039 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,043 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,146 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,147 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,152 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,253 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,253 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,258 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,359 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,359 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,364 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,387 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 19:41:03,465 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,465 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,470 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,571 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,571 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,576 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,677 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,677 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,682 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,783 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,784 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,790 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,891 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,892 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:03,897 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:03,910 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5899564e to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:03,913 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d579014, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:03,914 DEBUG [hconnection-0x33174c34-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:03,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:03,998 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:03,998 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,003 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,104 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,104 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,109 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,210 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,210 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,215 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,316 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,317 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,322 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,423 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,423 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,429 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,530 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,530 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,536 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,637 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,637 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,642 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,744 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,744 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,749 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,850 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,851 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,857 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:04,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x0a91992b to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:04,910 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ad13a38, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:04,912 DEBUG [hconnection-0x63d1d5a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:04,913 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:04,958 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:04,958 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:04,962 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,063 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,064 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,070 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,171 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,171 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,177 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,278 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,278 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,284 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,385 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,385 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,390 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,492 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,492 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,497 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,599 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,599 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,605 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,706 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,706 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,712 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,814 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,814 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,820 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:05,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x468d0972 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:05,911 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6afcfb52, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:05,913 DEBUG [hconnection-0x3bfd3726-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:05,915 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:05,921 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:05,921 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:05,929 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,030 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,030 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,036 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,137 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,138 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,144 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,246 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,246 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,252 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,354 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,354 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,360 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,462 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,462 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,468 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,570 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,570 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,577 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,679 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,679 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,687 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,789 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,789 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,797 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,898 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:06,898 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:06,906 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:06,953 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x52b4c96c to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:06,956 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46f7e11b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:06,958 DEBUG [hconnection-0x73abe2ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:06,959 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36918, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:07,019 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,019 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5515791f to 127.0.0.1:57973 2023-07-12 19:41:07,019 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,020 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x468d0972 to 127.0.0.1:57973 2023-07-12 19:41:07,020 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,020 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a91992b to 127.0.0.1:57973 2023-07-12 19:41:07,020 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,020 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x776809ad to 127.0.0.1:57973 2023-07-12 19:41:07,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a67f4d8 to 127.0.0.1:57973 2023-07-12 19:41:07,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x430d31d4 to 127.0.0.1:57973 2023-07-12 19:41:07,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x079a2ca0 to 127.0.0.1:57973 2023-07-12 19:41:07,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ba0d5c2 to 127.0.0.1:57973 2023-07-12 19:41:07,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12fd445d to 127.0.0.1:57973 2023-07-12 19:41:07,021 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,021 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e1c1974 to 127.0.0.1:57973 2023-07-12 19:41:07,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x120b1f55 to 127.0.0.1:57973 2023-07-12 19:41:07,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x770729cc to 127.0.0.1:57973 2023-07-12 19:41:07,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43e43ce3 to 127.0.0.1:57973 2023-07-12 19:41:07,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51356010 to 127.0.0.1:57973 2023-07-12 19:41:07,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x792476e5 to 127.0.0.1:57973 2023-07-12 19:41:07,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,022 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2acb1b19 to 127.0.0.1:57973 2023-07-12 19:41:07,022 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,023 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37698ef5 to 127.0.0.1:57973 2023-07-12 19:41:07,023 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,023 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b350a to 127.0.0.1:57973 2023-07-12 19:41:07,023 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,023 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29b5faf3 to 127.0.0.1:57973 2023-07-12 19:41:07,023 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,023 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x245344ed to 127.0.0.1:57973 2023-07-12 19:41:07,023 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,023 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01a78a98 to 127.0.0.1:57973 2023-07-12 19:41:07,023 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a604828 to 127.0.0.1:57973 2023-07-12 19:41:07,024 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7245998f to 127.0.0.1:57973 2023-07-12 19:41:07,024 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b5a74cf to 127.0.0.1:57973 2023-07-12 19:41:07,024 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5899564e to 127.0.0.1:57973 2023-07-12 19:41:07,024 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,024 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4399f2e9 to 127.0.0.1:57973 2023-07-12 19:41:07,024 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:07,032 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,134 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,134 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,140 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,242 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,242 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,248 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,349 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,349 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,357 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,458 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,459 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,465 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,567 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,567 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,573 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,674 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,675 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,680 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,781 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,782 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,791 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,893 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:07,893 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:07,899 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:07,912 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x18c57979 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:07,916 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@428e8adc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:07,919 DEBUG [hconnection-0x2e05d78a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:07,920 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36928, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:08,000 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,001 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,017 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,118 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,118 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,128 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,229 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,229 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,236 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,337 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,338 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,346 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,448 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,448 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,456 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,557 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,558 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,565 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,666 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,666 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,673 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,774 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,775 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,783 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,884 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,884 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:08,893 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:08,986 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x519995b4 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:08,989 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fd930fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:08,990 DEBUG [hconnection-0x78a6e907-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:08,992 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:08,995 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:08,995 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,003 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,104 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,105 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,112 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,213 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,213 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,219 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,321 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,321 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,327 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,429 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,429 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,435 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,536 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,537 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,543 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,644 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,644 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,651 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,752 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,752 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,759 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,860 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,861 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,867 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:09,911 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1d240d89 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:09,914 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25cfa71e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:09,915 DEBUG [hconnection-0x9be0c1a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:09,916 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36938, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:09,969 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:09,969 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:09,977 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,078 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,078 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,086 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,187 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,187 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,194 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,295 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,295 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,304 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,405 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,405 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,413 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,515 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,515 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,522 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,623 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,623 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,630 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,731 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,732 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,739 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,841 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,841 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,849 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:10,909 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x107745b1 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:10,912 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b7229e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:10,913 DEBUG [hconnection-0x75c16340-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:10,914 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52726, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:10,950 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:10,950 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:10,958 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,059 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,060 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,068 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,169 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,169 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,177 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,278 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,278 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,287 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,388 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,389 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,398 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,499 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,500 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,508 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,609 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,609 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,617 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,718 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,718 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,725 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,826 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,827 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,834 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:11,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe3e9bd to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:11,910 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@725c797f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:11,911 DEBUG [hconnection-0x4f2a2b34-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:11,913 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52736, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:11,935 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:11,935 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:11,945 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,046 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,046 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,055 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,156 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,156 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,164 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,266 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,266 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,275 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,376 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,377 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,389 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,490 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,491 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,502 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,603 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,604 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,614 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,715 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,715 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,724 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,825 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,825 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,834 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x171cce24 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:12,912 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21ac4f83, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:12,913 DEBUG [hconnection-0x6982095e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:12,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52748, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:12,935 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:12,935 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:12,945 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:12,948 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/5ee47a509c6444c78c697978f661c8fa] to archive 2023-07-12 19:41:12,951 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 19:41:12,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/5ee47a509c6444c78c697978f661c8fa to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/5ee47a509c6444c78c697978f661c8fa 2023-07-12 19:41:13,005 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testBucketingFilesToSnapshots7 moved into observance of table space quota. 2023-07-12 19:41:13,046 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$11(448): Waiting to see quota snapshot1 size 2023-07-12 19:41:13,046 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:13,055 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot1 references file=5ee47a509c6444c78c697978f661c8fa, size=297930 2023-07-12 19:41:13,056 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore(462): Snapshotting table again 2023-07-12 19:41:13,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } 2023-07-12 19:41:13,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190873057 (current time:1689190873057). 2023-07-12 19:41:13,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:41:13,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x0ddf0c6e to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:13,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c628ca1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:13,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:13,062 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52764, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:13,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ddf0c6e to 127.0.0.1:57973 2023-07-12 19:41:13,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:13,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:41:13,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } 2023-07-12 19:41:13,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=69, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE 2023-07-12 19:41:13,067 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=69, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE 2023-07-12 19:41:13,067 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=69, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE; timeout=600000, timestamp=1689191473067 2023-07-12 19:41:13,067 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testBucketingFilesToSnapshots7snapshot2 C_M_SNAPSHOT_TABLE on table testBucketingFilesToSnapshots7 2023-07-12 19:41:13,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } 2023-07-12 19:41:13,068 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:41:13,068 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:41:13,068 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=70, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED 2023-07-12 19:41:13,068 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=69, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE 2023-07-12 19:41:13,069 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=69, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=EXCLUSIVE in 4 msec 2023-07-12 19:41:13,070 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=70, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED 2023-07-12 19:41:13,070 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=70, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED; timeout=600000, timestamp=1689191473070 2023-07-12 19:41:13,168 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:13,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:13,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:41:13,169 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:41:13,370 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:13,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:13,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:41:13,371 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:41:13,478 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,478 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testBucketingFilesToSnapshots7snapshot2' 2023-07-12 19:41:13,478 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:41:13,478 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testBucketingFilesToSnapshots7snapshot2' starting 'acquire' 2023-07-12 19:41:13,478 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testBucketingFilesToSnapshots7snapshot2', kicking off acquire phase on members. 2023-07-12 19:41:13,478 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,478 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,479 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:41:13,479 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:41:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:41:13,480 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,480 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:41:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 90 2023-07-12 19:41:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,481 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testBucketingFilesToSnapshots7snapshot2 from table testBucketingFilesToSnapshots7 type SKIPFLUSH 2023-07-12 19:41:13,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,481 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testBucketingFilesToSnapshots7snapshot2' with timeout 300000ms 2023-07-12 19:41:13,481 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:41:13,482 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testBucketingFilesToSnapshots7snapshot2' starting 'acquire' stage 2023-07-12 19:41:13,482 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testBucketingFilesToSnapshots7snapshot2' locally acquired 2023-07-12 19:41:13,482 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testBucketingFilesToSnapshots7snapshot2) in zk 2023-07-12 19:41:13,485 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,485 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:41:13,485 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,485 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testBucketingFilesToSnapshots7snapshot2' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:41:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:13,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testBucketingFilesToSnapshots7snapshot2' on coordinator 2023-07-12 19:41:13,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f2cedc[Count = 0] remaining members to acquire global barrier 2023-07-12 19:41:13,486 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testBucketingFilesToSnapshots7snapshot2' starting 'in-barrier' execution. 2023-07-12 19:41:13,487 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,488 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testBucketingFilesToSnapshots7snapshot2' received 'reached' from coordinator. 2023-07-12 19:41:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:41:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:41:13,488 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:13,488 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,488 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:41:13,488 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:41:13,489 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa.' region-info for snapshot=testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,489 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:41:13,489 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/aaf3f083f9744e24a155f9acefc9dc7b] hfiles 2023-07-12 19:41:13,489 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/aaf3f083f9744e24a155f9acefc9dc7b for snapshot=testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,671 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:13,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:13,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:41:13,672 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:41:13,895 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. completed. 2023-07-12 19:41:13,895 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:13,895 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:41:13,895 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:41:13,895 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,895 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testBucketingFilesToSnapshots7snapshot2' locally completed 2023-07-12 19:41:13,895 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testBucketingFilesToSnapshots7snapshot2' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:41:13,897 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,897 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testBucketingFilesToSnapshots7snapshot2' has notified controller of completion 2023-07-12 19:41:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:41:13,897 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:41:13,897 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testBucketingFilesToSnapshots7snapshot2' completed. 2023-07-12 19:41:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testBucketingFilesToSnapshots7snapshot2' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:41:13,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testBucketingFilesToSnapshots7snapshot2', counting down latch. Waiting for 0 more 2023-07-12 19:41:13,899 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testBucketingFilesToSnapshots7snapshot2' execution completed 2023-07-12 19:41:13,899 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:41:13,899 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:41:13,899 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,899 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testBucketingFilesToSnapshots7snapshot2including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:41:13,901 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,901 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:41:13,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,901 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:41:13,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:13,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:41:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:41:13,902 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,903 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:13,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,905 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,905 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:41:13,905 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,905 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:41:13,905 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,905 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:41:13,905 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:41:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:41:13,905 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,906 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:13,906 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,906 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,906 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:41:13,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:41:13,906 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:13,911 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x40e038fe to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:13,917 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ded615f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:13,919 DEBUG [hconnection-0x5020bc57-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:13,920 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:13,928 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testBucketingFilesToSnapshots7snapshot2 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:14,172 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:14,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:14,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:41:14,174 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:41:14,753 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testBucketingFilesToSnapshots7snapshot2 of table testBucketingFilesToSnapshots7 completed 2023-07-12 19:41:14,753 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:14,753 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testBucketingFilesToSnapshots7snapshot2 2023-07-12 19:41:14,754 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testBucketingFilesToSnapshots7snapshot2 C_M_SNAPSHOT_TABLE on table testBucketingFilesToSnapshots7 at 1689190873067Consolidate snapshot: testBucketingFilesToSnapshots7snapshot2 at 1689190873905 (+838 ms)Loading Region manifests for testBucketingFilesToSnapshots7snapshot2 at 1689190873906 (+1 ms)Writing data manifest for testBucketingFilesToSnapshots7snapshot2 at 1689190873911 (+5 ms)Verifying snapshot: testBucketingFilesToSnapshots7snapshot2 at 1689190873919 (+8 ms)Snapshot testBucketingFilesToSnapshots7snapshot2 of table testBucketingFilesToSnapshots7 completed at 1689190874753 (+834 ms) 2023-07-12 19:41:14,755 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=70, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED 2023-07-12 19:41:14,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=70, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testBucketingFilesToSnapshots7, type=SHARED in 1.6870 sec 2023-07-12 19:41:14,945 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x33a3e11a to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:14,949 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ba38ae8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:14,954 DEBUG [hconnection-0x5a0c2a9e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:14,955 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52778, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:15,174 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:15,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testBucketingFilesToSnapshots7snapshot2 table=testBucketingFilesToSnapshots7 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:41:15,175 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore(466): Compacting table 2023-07-12 19:41:15,175 DEBUG [Listener at localhost/32953] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-07-12 19:41:15,175 DEBUG [Listener at localhost/32953] regionserver.HStore(1912): af83dc5342faa5fe20a6e4dc7a578aaa/f1 is initiating major compaction (all files) 2023-07-12 19:41:15,176 INFO [Listener at localhost/32953] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:41:15,176 INFO [Listener at localhost/32953] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:41:15,176 INFO [Listener at localhost/32953] regionserver.HRegion(2259): Starting compaction of af83dc5342faa5fe20a6e4dc7a578aaa/f1 in testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:15,176 INFO [Listener at localhost/32953] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/aaf3f083f9744e24a155f9acefc9dc7b] into tmpdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/.tmp, totalSize=291.0 K 2023-07-12 19:41:15,177 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting aaf3f083f9744e24a155f9acefc9dc7b, keycount=1024, bloomtype=ROW, size=291.0 K, encoding=NONE, compression=NONE, seqNum=25, earliestPutTs=1689190858815 2023-07-12 19:41:15,187 INFO [Listener at localhost/32953] throttle.PressureAwareThroughputController(145): af83dc5342faa5fe20a6e4dc7a578aaa#f1#compaction#14 average throughput is 46.05 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 19:41:15,598 DEBUG [Listener at localhost/32953] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/.tmp/f1/d241cc55d4094f10b9d8e3fc04b37f87 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/d241cc55d4094f10b9d8e3fc04b37f87 2023-07-12 19:41:15,602 INFO [Listener at localhost/32953] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in af83dc5342faa5fe20a6e4dc7a578aaa/f1 of af83dc5342faa5fe20a6e4dc7a578aaa into d241cc55d4094f10b9d8e3fc04b37f87(size=291.0 K), total size for store is 291.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 19:41:15,603 DEBUG [Listener at localhost/32953] regionserver.HRegion(2289): Compaction status journal for af83dc5342faa5fe20a6e4dc7a578aaa: 2023-07-12 19:41:15,603 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:41:15,603 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:15,603 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:15,611 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:15,713 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:15,713 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:15,721 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:15,823 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:15,823 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:15,831 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:15,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x10b0e889 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:15,910 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@715d12db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:15,911 DEBUG [hconnection-0x768f8aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:15,913 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:15,932 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:15,933 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:15,942 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,043 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,043 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,052 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,153 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,154 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,163 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,264 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,264 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,273 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,374 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,374 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,383 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,484 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,484 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,496 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,597 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,597 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,606 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,707 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,708 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,717 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,818 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,818 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,827 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,928 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:16,929 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:16,940 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:16,941 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x46cb72d0 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:16,944 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71877301, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:16,945 DEBUG [hconnection-0x5b153359-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:16,947 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:17,041 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,042 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,052 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:17,152 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,153 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,162 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:17,264 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,264 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,275 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:17,376 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,376 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,388 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:17,489 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,489 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,499 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:17,600 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,600 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,610 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:17,711 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,711 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,723 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:17,825 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,825 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,840 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:17,909 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x6c53b7e5 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:17,915 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3fa42ef8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:17,917 DEBUG [hconnection-0x65faa5f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:17,918 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52800, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:17,941 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:17,941 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:17,957 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,058 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,059 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:18,072 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,173 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,173 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:18,190 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,304 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,305 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:18,318 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,419 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,419 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:18,435 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,536 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,536 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:18,549 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,650 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,651 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:18,667 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,769 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,769 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:18,783 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,884 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,884 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:18,897 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:18,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x6bff626d to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:18,911 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ab51e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:18,912 DEBUG [hconnection-0x20753d21-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:18,913 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52802, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:18,998 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:18,998 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,016 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:19,117 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:19,117 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,131 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:19,233 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:19,233 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,249 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:19,350 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:19,351 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,367 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:19,468 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:19,468 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,484 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:19,586 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:19,586 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,599 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:19,700 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:19,700 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,712 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:19,813 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:19,813 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,824 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:19,916 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x6b99c4df to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:19,919 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7645fd18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:19,921 DEBUG [hconnection-0x539cefce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:19,922 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:19,925 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:19,925 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:19,941 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,042 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,042 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,062 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,164 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,164 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,183 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,230 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x107745b1 to 127.0.0.1:57973 2023-07-12 19:41:20,231 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,231 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b99c4df to 127.0.0.1:57973 2023-07-12 19:41:20,231 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,231 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6bff626d to 127.0.0.1:57973 2023-07-12 19:41:20,231 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,231 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x519995b4 to 127.0.0.1:57973 2023-07-12 19:41:20,231 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,231 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18c57979 to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52b4c96c to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d240d89 to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe3e9bd to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33a3e11a to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e038fe to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x171cce24 to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46cb72d0 to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10b0e889 to 127.0.0.1:57973 2023-07-12 19:41:20,232 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,233 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c53b7e5 to 127.0.0.1:57973 2023-07-12 19:41:20,233 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:20,284 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,284 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,297 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,398 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,398 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,412 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,519 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,519 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,532 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,633 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,634 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,645 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,746 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,747 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,758 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,859 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,859 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,871 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:20,913 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7edbbe09 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:20,916 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2dd2f7cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:20,917 DEBUG [hconnection-0x6e0e01e1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:20,918 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:20,972 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:20,972 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:20,987 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,088 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:21,088 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:21,101 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,202 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:21,203 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:21,215 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,316 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:21,317 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:21,332 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,433 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:21,433 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:21,450 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,551 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:21,551 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:21,563 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,664 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:21,664 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:21,676 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,777 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:21,777 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:21,789 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,890 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:21,890 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:21,902 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:21,909 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5ac0a2d0 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:21,913 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58421b1e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:21,914 DEBUG [hconnection-0x95cac7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:21,915 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:22,003 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,004 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,022 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,123 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,124 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,136 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,237 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,238 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,250 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,351 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,352 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,364 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,465 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,465 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,478 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,579 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,579 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,592 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,693 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,693 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,706 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,807 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,808 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,820 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,921 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:22,921 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:22,934 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:22,987 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x672d0b5f to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:22,992 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5fbca781, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:22,993 DEBUG [hconnection-0x7a6867c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:22,994 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:23,035 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,036 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,052 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:23,153 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,153 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,166 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:23,267 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,267 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,280 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:23,381 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,382 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,395 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:23,496 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,496 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,510 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:23,611 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,611 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,625 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:23,726 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,726 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,740 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:23,841 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,841 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,855 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:23,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1ad3f8a1 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:23,910 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6721d03f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:23,912 DEBUG [hconnection-0x4e0b97b9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:23,913 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:23,956 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:23,956 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:23,972 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,073 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,073 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:24,088 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,189 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,189 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:24,203 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,304 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,304 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:24,318 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,419 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,420 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:24,434 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,535 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,535 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:24,549 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,650 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,651 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:24,665 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,766 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,766 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:24,781 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,882 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,882 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:24,896 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:24,940 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x31e45b30 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:24,944 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30f0e42c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:24,945 DEBUG [hconnection-0x41ffc407-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:24,946 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:24,997 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:24,998 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,024 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:25,125 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:25,125 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,140 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:25,241 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:25,241 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,255 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:25,356 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:25,357 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,371 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:25,472 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:25,472 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,487 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:25,588 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:25,588 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,603 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:25,704 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:25,704 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,719 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:25,820 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:25,820 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,843 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:25,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2b84fd2d to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:25,910 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56153a10, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:25,912 DEBUG [hconnection-0x607a9869-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:25,913 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:25,944 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:25,945 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:25,973 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,074 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:26,075 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:26,096 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,197 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:26,198 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:26,220 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,321 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:26,321 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:26,337 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,438 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:26,439 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:26,454 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,555 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:26,555 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:26,571 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,672 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:26,672 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:26,688 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,788 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:26,789 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:26,805 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,906 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:26,906 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:26,922 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:26,924 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x10406273 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:26,927 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@413fc4da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:26,928 DEBUG [hconnection-0x14076a4e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:26,929 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:27,023 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,024 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,041 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,142 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,142 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,158 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,259 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,259 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,275 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,376 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,376 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,392 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,493 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,494 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,510 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,610 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,611 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,627 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,728 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,728 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,744 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,845 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,846 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,862 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,910 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x17608a71 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:27,912 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2050be41, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:27,914 DEBUG [hconnection-0xd680be1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:27,914 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:27,963 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:27,963 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:27,986 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:27,992 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/aaf3f083f9744e24a155f9acefc9dc7b] to archive 2023-07-12 19:41:27,997 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 19:41:27,999 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/aaf3f083f9744e24a155f9acefc9dc7b to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/f1/aaf3f083f9744e24a155f9acefc9dc7b 2023-07-12 19:41:28,087 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$12(474): Waiting to see quota snapshot2 size 2023-07-12 19:41:28,087 INFO [Listener at localhost/32953] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:41:28,115 INFO [Listener at localhost/32953] quotas.TestSnapshotQuotaObserverChore$13(510): Snapshot=testBucketingFilesToSnapshots7snapshot2 references file=aaf3f083f9744e24a155f9acefc9dc7b, size=297964 2023-07-12 19:41:28,128 INFO [Listener at localhost/32953] hbase.ResourceChecker(175): after: quotas.TestSnapshotQuotaObserverChore#testBucketingFilesToSnapshots Thread=251 (was 301), OpenFileDescriptor=533 (was 617), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=213 (was 122) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=7999 (was 7985) - AvailableMemoryMB LEAK? - 2023-07-12 19:41:28,139 INFO [Listener at localhost/32953] hbase.ResourceChecker(147): before: quotas.TestSnapshotQuotaObserverChore#testRemovedSnapshots Thread=251, OpenFileDescriptor=533, MaxFileDescriptor=60000, SystemLoadAverage=213, ProcessCount=172, AvailableMemoryMB=7998 2023-07-12 19:41:28,140 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(139): Connect 0x1e0d213a to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:28,143 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15f511dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:28,144 DEBUG [hconnection-0x3ac6f2d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:28,146 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:28,148 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(447): Deleting quota for table: testBucketingFilesToSnapshots7 2023-07-12 19:41:28,149 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e0d213a to 127.0.0.1:57973 2023-07-12 19:41:28,149 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:28,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testRemovedSnapshots8', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:41:28,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=71, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRemovedSnapshots8 2023-07-12 19:41:28,153 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=71, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRemovedSnapshots8 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:41:28,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRemovedSnapshots8" procId is: 71 2023-07-12 19:41:28,154 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=71, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRemovedSnapshots8 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:41:28,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=71 2023-07-12 19:41:28,158 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,158 DEBUG [HFileArchiver-9] backup.HFileArchiver(153): Directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1 empty. 2023-07-12 19:41:28,159 DEBUG [HFileArchiver-9] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,159 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRemovedSnapshots8 regions 2023-07-12 19:41:28,170 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp/data/default/testRemovedSnapshots8/.tabledesc/.tableinfo.0000000001 2023-07-12 19:41:28,171 INFO [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(7675): creating {ENCODED => de854fce7414330f8a87a4432783eca1, NAME => 'testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRemovedSnapshots8', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.tmp 2023-07-12 19:41:28,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=71 2023-07-12 19:41:28,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=71 2023-07-12 19:41:28,576 DEBUG [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(866): Instantiated testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:41:28,576 DEBUG [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(1604): Closing de854fce7414330f8a87a4432783eca1, disabling compactions & flushes 2023-07-12 19:41:28,576 INFO [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(1626): Closing region testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:28,576 DEBUG [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:28,576 DEBUG [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(1714): Acquired close lock on testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. after waiting 0 ms 2023-07-12 19:41:28,576 DEBUG [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(1724): Updates disabled for region testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:28,576 INFO [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(1838): Closed testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:28,576 DEBUG [RegionOpenAndInit-testRemovedSnapshots8-pool-0] regionserver.HRegion(1558): Region close journal for de854fce7414330f8a87a4432783eca1: 2023-07-12 19:41:28,579 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=71, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRemovedSnapshots8 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:41:28,580 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1.","families":{"info":[{"qualifier":"regioninfo","vlen":55,"tag":[],"timestamp":"1689190888579"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689190888579"}]},"ts":"1689190888579"} 2023-07-12 19:41:28,581 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:41:28,581 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=71, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRemovedSnapshots8 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:41:28,582 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRemovedSnapshots8","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190888581"}]},"ts":"1689190888581"} 2023-07-12 19:41:28,582 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRemovedSnapshots8, state=ENABLING in hbase:meta 2023-07-12 19:41:28,587 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=71, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRemovedSnapshots8, region=de854fce7414330f8a87a4432783eca1, ASSIGN}] 2023-07-12 19:41:28,588 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, ppid=71, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRemovedSnapshots8, region=de854fce7414330f8a87a4432783eca1, ASSIGN 2023-07-12 19:41:28,588 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=72, ppid=71, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRemovedSnapshots8, region=de854fce7414330f8a87a4432783eca1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44631,1689190811374; forceNewPlan=false, retain=false 2023-07-12 19:41:28,739 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=72 updating hbase:meta row=de854fce7414330f8a87a4432783eca1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:28,740 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1.","families":{"info":[{"qualifier":"regioninfo","vlen":55,"tag":[],"timestamp":"1689190888739"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689190888739"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689190888739"}]},"ts":"1689190888739"} 2023-07-12 19:41:28,741 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=72, state=RUNNABLE; OpenRegionProcedure de854fce7414330f8a87a4432783eca1, server=jenkins-hbase4.apache.org,44631,1689190811374}] 2023-07-12 19:41:28,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=71 2023-07-12 19:41:28,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:28,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => de854fce7414330f8a87a4432783eca1, NAME => 'testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:41:28,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRemovedSnapshots8 de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:41:28,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,897 INFO [StoreOpener-de854fce7414330f8a87a4432783eca1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,898 DEBUG [StoreOpener-de854fce7414330f8a87a4432783eca1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1 2023-07-12 19:41:28,898 DEBUG [StoreOpener-de854fce7414330f8a87a4432783eca1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1 2023-07-12 19:41:28,898 INFO [StoreOpener-de854fce7414330f8a87a4432783eca1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region de854fce7414330f8a87a4432783eca1 columnFamilyName f1 2023-07-12 19:41:28,899 INFO [StoreOpener-de854fce7414330f8a87a4432783eca1-1] regionserver.HStore(310): Store=de854fce7414330f8a87a4432783eca1/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:41:28,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:28,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:41:28,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened de854fce7414330f8a87a4432783eca1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72835542, jitterRate=0.08533415198326111}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:41:28,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for de854fce7414330f8a87a4432783eca1: 2023-07-12 19:41:28,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1., pid=73, masterSystemTime=1689190888893 2023-07-12 19:41:28,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:28,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:28,906 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=72 updating hbase:meta row=de854fce7414330f8a87a4432783eca1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:28,907 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1.","families":{"info":[{"qualifier":"regioninfo","vlen":55,"tag":[],"timestamp":"1689190888906"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689190888906"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689190888906"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689190888906"}]},"ts":"1689190888906"} 2023-07-12 19:41:28,908 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x29ddf4b0 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:28,909 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=72 2023-07-12 19:41:28,909 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=72, state=SUCCESS; OpenRegionProcedure de854fce7414330f8a87a4432783eca1, server=jenkins-hbase4.apache.org,44631,1689190811374 in 167 msec 2023-07-12 19:41:28,910 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4df950fe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:28,911 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=71 2023-07-12 19:41:28,911 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=71, state=SUCCESS; TransitRegionStateProcedure table=testRemovedSnapshots8, region=de854fce7414330f8a87a4432783eca1, ASSIGN in 322 msec 2023-07-12 19:41:28,911 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=71, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRemovedSnapshots8 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:41:28,911 DEBUG [hconnection-0x66ac6bd2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:28,912 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRemovedSnapshots8","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689190888912"}]},"ts":"1689190888912"} 2023-07-12 19:41:28,912 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:28,913 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRemovedSnapshots8, state=ENABLED in hbase:meta 2023-07-12 19:41:28,915 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=71, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRemovedSnapshots8 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:41:28,916 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=71, state=SUCCESS; CreateTableProcedure table=testRemovedSnapshots8 in 763 msec 2023-07-12 19:41:29,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1230): Checking to see if procedure is done pid=71 2023-07-12 19:41:29,259 INFO [Listener at localhost/32953] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRemovedSnapshots8, procId: 71 completed 2023-07-12 19:41:29,305 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 19:41:29,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(936): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-07-12 19:41:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure.ProcedureCoordinator(165): Submitting procedure testRemovedSnapshots8 2023-07-12 19:41:29,315 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRemovedSnapshots8' 2023-07-12 19:41:29,315 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:41:29,315 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRemovedSnapshots8' starting 'acquire' 2023-07-12 19:41:29,315 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRemovedSnapshots8', kicking off acquire phase on members. 2023-07-12 19:41:29,316 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,316 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRemovedSnapshots8 2023-07-12 19:41:29,318 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,318 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:41:29,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:41:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:41:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRemovedSnapshots8 2023-07-12 19:41:29,319 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,319 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:41:29,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 19:41:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRemovedSnapshots8 2023-07-12 19:41:29,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRemovedSnapshots8 2023-07-12 19:41:29,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRemovedSnapshots8 2023-07-12 19:41:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRemovedSnapshots8' with timeout 60000ms 2023-07-12 19:41:29,320 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 19:41:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRemovedSnapshots8' starting 'acquire' stage 2023-07-12 19:41:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 19:41:29,321 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 19:41:29,321 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:29,321 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. started... 2023-07-12 19:41:29,321 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing de854fce7414330f8a87a4432783eca1 1/1 column families, dataSize=282.92 KB heapSize=360.25 KB 2023-07-12 19:41:29,734 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=282.92 KB at sequenceid=25 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/.tmp/f1/96089a295062431bae6f8b14b7816677 2023-07-12 19:41:29,739 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/.tmp/f1/96089a295062431bae6f8b14b7816677 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/96089a295062431bae6f8b14b7816677 2023-07-12 19:41:29,743 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/96089a295062431bae6f8b14b7816677, entries=1024, sequenceid=25, filesize=290.9 K 2023-07-12 19:41:29,744 INFO [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~282.92 KB/289706, heapSize ~360.23 KB/368880, currentSize=0 B/0 for de854fce7414330f8a87a4432783eca1 in 423ms, sequenceid=25, compaction requested=false 2023-07-12 19:41:29,744 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRemovedSnapshots8' 2023-07-12 19:41:29,745 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for de854fce7414330f8a87a4432783eca1: 2023-07-12 19:41:29,745 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 19:41:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 19:41:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRemovedSnapshots8' locally acquired 2023-07-12 19:41:29,745 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testRemovedSnapshots8) in zk 2023-07-12 19:41:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRemovedSnapshots8 2023-07-12 19:41:29,746 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:41:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRemovedSnapshots8 2023-07-12 19:41:29,747 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRemovedSnapshots8' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:41:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8 2023-07-12 19:41:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:29,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testRemovedSnapshots8' on coordinator 2023-07-12 19:41:29,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7736b11f[Count = 0] remaining members to acquire global barrier 2023-07-12 19:41:29,748 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRemovedSnapshots8' starting 'in-barrier' execution. 2023-07-12 19:41:29,748 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRemovedSnapshots8 2023-07-12 19:41:29,749 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRemovedSnapshots8 2023-07-12 19:41:29,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRemovedSnapshots8 2023-07-12 19:41:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRemovedSnapshots8 2023-07-12 19:41:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRemovedSnapshots8' received 'reached' from coordinator. 2023-07-12 19:41:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRemovedSnapshots8' locally completed 2023-07-12 19:41:29,749 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,749 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:41:29,749 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRemovedSnapshots8' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:41:29,750 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRemovedSnapshots8' has notified controller of completion 2023-07-12 19:41:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:41:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:41:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRemovedSnapshots8' completed. 2023-07-12 19:41:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8 2023-07-12 19:41:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8 2023-07-12 19:41:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRemovedSnapshots8' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:41:29,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testRemovedSnapshots8', counting down latch. Waiting for 0 more 2023-07-12 19:41:29,753 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRemovedSnapshots8' execution completed 2023-07-12 19:41:29,753 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:41:29,753 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:41:29,753 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRemovedSnapshots8 2023-07-12 19:41:29,753 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRemovedSnapshots8including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 19:41:29,756 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,756 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 19:41:29,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,756 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:41:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:29,756 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:41:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:41:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8 2023-07-12 19:41:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8 2023-07-12 19:41:29,757 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8 2023-07-12 19:41:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,759 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,759 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 19:41:29,759 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRemovedSnapshots8 2023-07-12 19:41:29,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRemovedSnapshots8' 2023-07-12 19:41:29,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 19:41:29,759 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRemovedSnapshots8 2023-07-12 19:41:29,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 19:41:29,759 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:41:29,759 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 19:41:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:41:29,760 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRemovedSnapshots8/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:29,760 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRemovedSnapshots8'' to complete. (max 20000 ms per retry) 2023-07-12 19:41:29,760 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 19:41:29,760 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRemovedSnapshots8 2023-07-12 19:41:29,760 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRemovedSnapshots8 2023-07-12 19:41:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 19:41:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:41:29,760 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRemovedSnapshots8 2023-07-12 19:41:29,860 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 19:41:29,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 19:41:29,861 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 19:41:29,862 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:41:29,863 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: null 2023-07-12 19:41:29,910 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testRemovedSnapshots8 moved into observance of table space quota. 2023-07-12 19:41:29,911 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x54ffa64f to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:29,914 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a5501a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:29,915 DEBUG [hconnection-0x7d8f6fa9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:29,916 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:29,976 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17608a71 to 127.0.0.1:57973 2023-07-12 19:41:29,976 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:29,976 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7edbbe09 to 127.0.0.1:57973 2023-07-12 19:41:29,976 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:29,976 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ac0a2d0 to 127.0.0.1:57973 2023-07-12 19:41:29,976 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:29,976 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ad3f8a1 to 127.0.0.1:57973 2023-07-12 19:41:29,976 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:29,976 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x672d0b5f to 127.0.0.1:57973 2023-07-12 19:41:29,976 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:29,976 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31e45b30 to 127.0.0.1:57973 2023-07-12 19:41:29,976 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:29,977 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10406273 to 127.0.0.1:57973 2023-07-12 19:41:29,977 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:29,977 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b84fd2d to 127.0.0.1:57973 2023-07-12 19:41:29,977 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:29,977 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:29,977 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29ddf4b0 to 127.0.0.1:57973 2023-07-12 19:41:29,977 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:30,078 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,180 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,281 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,382 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,483 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,584 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,686 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,787 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,888 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:30,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x374fea4c to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:30,912 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14cd18c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:30,913 DEBUG [hconnection-0x485a3a3f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:30,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:30,990 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 19:41:31,048 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testRemovedSnapshots8 moved into observance of table space quota. 2023-07-12 19:41:31,092 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=290.95 KB/1 GB] 2023-07-12 19:41:31,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1583): Client=jenkins//172.31.14.131 snapshot request for:{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 } 2023-07-12 19:41:31,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689190891093 (current time:1689190891093). 2023-07-12 19:41:31,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 19:41:31,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(139): Connect 0x4ce543af to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:31,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@323aae5e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:31,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:31,099 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:31,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ce543af to 127.0.0.1:57973 2023-07-12 19:41:31,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:31,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 19:41:31,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 } 2023-07-12 19:41:31,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] procedure2.ProcedureExecutor(1029): Stored pid=74, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=EXCLUSIVE 2023-07-12 19:41:31,103 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=74, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=EXCLUSIVE 2023-07-12 19:41:31,104 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=74, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=EXCLUSIVE; timeout=600000, timestamp=1689191491104 2023-07-12 19:41:31,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(653): Started snapshot: { ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 } 2023-07-12 19:41:31,104 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running SKIPFLUSH table snapshot testRemovedSnapshots8snapshot1 C_M_SNAPSHOT_TABLE on table testRemovedSnapshots8 2023-07-12 19:41:31,105 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 19:41:31,105 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 19:41:31,105 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=74, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=EXCLUSIVE 2023-07-12 19:41:31,105 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure2.ProcedureExecutor(1029): Stored pid=75, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=SHARED 2023-07-12 19:41:31,106 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=74, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=EXCLUSIVE in 3 msec 2023-07-12 19:41:31,106 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=75, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=SHARED 2023-07-12 19:41:31,107 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=75, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=SHARED; timeout=600000, timestamp=1689191491107 2023-07-12 19:41:31,205 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:31,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:31,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:41:31,206 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 19:41:31,406 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:31,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:31,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:41:31,408 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 19:41:31,513 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,514 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRemovedSnapshots8snapshot1' 2023-07-12 19:41:31,514 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:41:31,514 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRemovedSnapshots8snapshot1' starting 'acquire' 2023-07-12 19:41:31,514 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRemovedSnapshots8snapshot1', kicking off acquire phase on members. 2023-07-12 19:41:31,514 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,514 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,515 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,515 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:41:31,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:41:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:41:31,516 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,516 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 19:41:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 72 2023-07-12 19:41:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,517 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testRemovedSnapshots8snapshot1 from table testRemovedSnapshots8 type SKIPFLUSH 2023-07-12 19:41:31,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,517 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRemovedSnapshots8snapshot1' with timeout 300000ms 2023-07-12 19:41:31,517 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 19:41:31,517 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRemovedSnapshots8snapshot1' starting 'acquire' stage 2023-07-12 19:41:31,517 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRemovedSnapshots8snapshot1' locally acquired 2023-07-12 19:41:31,517 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure (testRemovedSnapshots8snapshot1) in zk 2023-07-12 19:41:31,519 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:41:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRemovedSnapshots8snapshot1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 19:41:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:31,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44631,1689190811374' joining acquired barrier for procedure 'testRemovedSnapshots8snapshot1' on coordinator 2023-07-12 19:41:31,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c0e52d[Count = 0] remaining members to acquire global barrier 2023-07-12 19:41:31,521 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRemovedSnapshots8snapshot1' starting 'in-barrier' execution. 2023-07-12 19:41:31,521 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,523 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,523 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRemovedSnapshots8snapshot1' received 'reached' from coordinator. 2023-07-12 19:41:31,523 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,523 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 19:41:31,523 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 19:41:31,523 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 19:41:31,523 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:31,523 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(108): take snapshot without flush memstore first 2023-07-12 19:41:31,523 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1.' region-info for snapshot=testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,524 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 19:41:31,524 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/96089a295062431bae6f8b14b7816677] hfiles 2023-07-12 19:41:31,524 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/96089a295062431bae6f8b14b7816677 for snapshot=testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,708 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:41:31,709 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 19:41:31,912 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4dbe8b35 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:31,915 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74bc04a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:31,916 DEBUG [hconnection-0x63d13b63-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:31,917 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56694, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:31,929 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(135): ... SkipFlush Snapshotting region testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. completed. 2023-07-12 19:41:31,929 DEBUG [rs(jenkins-hbase4.apache.org,44631,1689190811374)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:31,929 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 19:41:31,929 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 19:41:31,929 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,929 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRemovedSnapshots8snapshot1' locally completed 2023-07-12 19:41:31,929 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRemovedSnapshots8snapshot1' completed for member 'jenkins-hbase4.apache.org,44631,1689190811374' in zk 2023-07-12 19:41:31,931 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,931 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRemovedSnapshots8snapshot1' has notified controller of completion 2023-07-12 19:41:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:41:31,931 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:41:31,931 DEBUG [member: 'jenkins-hbase4.apache.org,44631,1689190811374' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRemovedSnapshots8snapshot1' completed. 2023-07-12 19:41:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRemovedSnapshots8snapshot1' member 'jenkins-hbase4.apache.org,44631,1689190811374': 2023-07-12 19:41:31,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44631,1689190811374' released barrier for procedure'testRemovedSnapshots8snapshot1', counting down latch. Waiting for 0 more 2023-07-12 19:41:31,933 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRemovedSnapshots8snapshot1' execution completed 2023-07-12 19:41:31,933 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 19:41:31,933 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 19:41:31,933 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,933 INFO [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRemovedSnapshots8snapshot1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 19:41:31,935 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,935 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 19:41:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,935 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:41:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 19:41:31,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:41:31,936 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:41:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 19:41:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 19:41:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,937 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 19:41:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,943 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,943 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 because node does not exist (not an error) 2023-07-12 19:41:31,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 19:41:31,943 DEBUG [(jenkins-hbase4.apache.org,33647,1689190810081)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 19:41:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,943 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,944 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testRemovedSnapshots8snapshot1 2023-07-12 19:41:31,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 19:41:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:41:31,983 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testRemovedSnapshots8snapshot1 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/testRemovedSnapshots8snapshot1 2023-07-12 19:41:32,209 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:32,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:32,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 }' is still in progress! 2023-07-12 19:41:32,210 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 19:41:32,819 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testRemovedSnapshots8snapshot1 of table testRemovedSnapshots8 completed 2023-07-12 19:41:32,819 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testRemovedSnapshots8snapshot1 2023-07-12 19:41:32,819 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/.hbase-snapshot/.tmp/testRemovedSnapshots8snapshot1 2023-07-12 19:41:32,819 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running SKIPFLUSH table snapshot testRemovedSnapshots8snapshot1 C_M_SNAPSHOT_TABLE on table testRemovedSnapshots8 at 1689190891104Consolidate snapshot: testRemovedSnapshots8snapshot1 at 1689190891943 (+839 ms)Loading Region manifests for testRemovedSnapshots8snapshot1 at 1689190891943Writing data manifest for testRemovedSnapshots8snapshot1 at 1689190891954 (+11 ms)Verifying snapshot: testRemovedSnapshots8snapshot1 at 1689190891959 (+5 ms)Snapshot testRemovedSnapshots8snapshot1 of table testRemovedSnapshots8 completed at 1689190892819 (+860 ms) 2023-07-12 19:41:32,820 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=75, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=SHARED 2023-07-12 19:41:32,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRemovedSnapshots8, type=SHARED in 1.7150 sec 2023-07-12 19:41:32,946 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x14f9cac1 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:32,949 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@399ad817, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:32,951 DEBUG [hconnection-0x1926498c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:32,952 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56704, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:33,210 DEBUG [Listener at localhost/32953] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 19:41:33,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 } is done 2023-07-12 19:41:33,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(401): Snapshot '{ ss=testRemovedSnapshots8snapshot1 table=testRemovedSnapshots8 type=SKIPFLUSH ttl=0 }' has completed, notifying client. 2023-07-12 19:41:33,213 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:41:33,214 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:41:33,215 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=290.95 KB/1 GB] 2023-07-12 19:41:33,215 DEBUG [Listener at localhost/32953] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-07-12 19:41:33,215 DEBUG [Listener at localhost/32953] regionserver.HStore(1912): de854fce7414330f8a87a4432783eca1/f1 is initiating major compaction (all files) 2023-07-12 19:41:33,216 INFO [Listener at localhost/32953] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:41:33,216 INFO [Listener at localhost/32953] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:41:33,216 INFO [Listener at localhost/32953] regionserver.HRegion(2259): Starting compaction of de854fce7414330f8a87a4432783eca1/f1 in testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:33,216 INFO [Listener at localhost/32953] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/96089a295062431bae6f8b14b7816677] into tmpdir=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/.tmp, totalSize=290.9 K 2023-07-12 19:41:33,217 DEBUG [Listener at localhost/32953] compactions.Compactor(207): Compacting 96089a295062431bae6f8b14b7816677, keycount=1024, bloomtype=ROW, size=290.9 K, encoding=NONE, compression=NONE, seqNum=25, earliestPutTs=1689190889264 2023-07-12 19:41:33,226 INFO [Listener at localhost/32953] throttle.PressureAwareThroughputController(145): de854fce7414330f8a87a4432783eca1#f1#compaction#16 average throughput is 55.26 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 19:41:33,240 DEBUG [Listener at localhost/32953] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/.tmp/f1/834dd9042cb84cd49980b39e786ab72a as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/834dd9042cb84cd49980b39e786ab72a 2023-07-12 19:41:33,246 INFO [Listener at localhost/32953] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in de854fce7414330f8a87a4432783eca1/f1 of de854fce7414330f8a87a4432783eca1 into 834dd9042cb84cd49980b39e786ab72a(size=291.0 K), total size for store is 291.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 19:41:33,246 DEBUG [Listener at localhost/32953] regionserver.HRegion(2289): Compaction status journal for de854fce7414330f8a87a4432783eca1: 2023-07-12 19:41:33,247 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:41:33,921 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x20457466 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:33,925 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13391508, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:33,926 DEBUG [hconnection-0x6286c5af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:33,927 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56714, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:34,424 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 19:41:34,946 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3ceff890 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:34,949 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b6fde9a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:34,950 DEBUG [hconnection-0x372be88c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:34,951 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56716, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:35,093 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testRemovedSnapshots8 moved into observance of table space quota. 2023-07-12 19:41:35,930 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x10ff733e to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:35,933 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e9a4f74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:35,934 DEBUG [hconnection-0x481c097-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:35,935 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56722, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:36,910 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7bf0df20 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:36,914 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b8a6de9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:36,915 DEBUG [hconnection-0x5cf68029-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:36,916 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:37,985 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x79cf84e1 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:37,987 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ee4ac58, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:37,989 DEBUG [hconnection-0x597f9b5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:37,990 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:38,909 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x41437606 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:38,912 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@785d08a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:38,914 DEBUG [hconnection-0x4afb91f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:38,914 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:39,910 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x63e7813b to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:39,912 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a30ada, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:39,913 DEBUG [hconnection-0x10a43573-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:39,914 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:40,910 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5c523f9d to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:40,913 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ac03835, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:40,914 DEBUG [hconnection-0x30792d2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:40,915 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35352, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:41,917 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5808dfd7 to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:41,920 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d25cad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:41,922 DEBUG [hconnection-0x674ad85e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:41,923 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:42,907 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1a506f0b to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:42,910 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ac84224, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:42,912 DEBUG [hconnection-0x15c4b749-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:42,913 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35358, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:42,948 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/96089a295062431bae6f8b14b7816677] to archive 2023-07-12 19:41:42,948 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 19:41:42,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/96089a295062431bae6f8b14b7816677 to hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/archive/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/f1/96089a295062431bae6f8b14b7816677 2023-07-12 19:41:43,039 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:41:43,040 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=290.98 KB/1 GB] 2023-07-12 19:41:43,083 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testRemovedSnapshots8 moved into observance of table space quota. 2023-07-12 19:41:43,142 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] master.MasterRpcServices(751): Client=jenkins//172.31.14.131 delete name: "testRemovedSnapshots8snapshot1" 2023-07-12 19:41:43,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33647] snapshot.SnapshotManager(333): Deleting snapshot: testRemovedSnapshots8snapshot1 2023-07-12 19:41:43,162 INFO [Listener at localhost/32953] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:41:43,163 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,264 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,365 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,466 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,568 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,669 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,770 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,871 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:43,922 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2b8ff6db to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:43,926 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ddf742c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:43,928 DEBUG [hconnection-0x7a8e71a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:43,929 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35374, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:43,973 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=581.93 KB/1 GB] 2023-07-12 19:41:44,040 DEBUG [master/jenkins-hbase4:0.Chore.1] quotas.QuotaObserverChore(336): testRemovedSnapshots8 moved into observance of table space quota. 2023-07-12 19:41:44,074 DEBUG [Listener at localhost/32953] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRemovedSnapshots8: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=290.98 KB/1 GB] 2023-07-12 19:41:44,088 INFO [Listener at localhost/32953] hbase.ResourceChecker(175): after: quotas.TestSnapshotQuotaObserverChore#testRemovedSnapshots Thread=272 (was 251) - Thread LEAK? -, OpenFileDescriptor=578 (was 533) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=166 (was 213), ProcessCount=172 (was 172), AvailableMemoryMB=7773 (was 7998) 2023-07-12 19:41:44,089 INFO [Listener at localhost/32953] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 19:41:44,089 INFO [Listener at localhost/32953] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 19:41:44,089 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x309d5bdc to 127.0.0.1:57973 2023-07-12 19:41:44,089 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:44,089 DEBUG [Listener at localhost/32953] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 19:41:44,089 DEBUG [Listener at localhost/32953] util.JVMClusterUtil(257): Found active master hash=1910381210, stopped=false 2023-07-12 19:41:44,089 DEBUG [Listener at localhost/32953] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.quotas.MasterQuotasObserver 2023-07-12 19:41:44,090 INFO [Listener at localhost/32953] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:41:44,091 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:41:44,091 INFO [Listener at localhost/32953] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:41:44,091 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:41:44,091 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:41:44,092 DEBUG [Listener at localhost/32953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40530357 to 127.0.0.1:57973 2023-07-12 19:41:44,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:41:44,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:41:44,092 DEBUG [Listener at localhost/32953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:44,092 INFO [Listener at localhost/32953] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,44631,1689190811374' ***** 2023-07-12 19:41:44,092 INFO [Listener at localhost/32953] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 19:41:44,092 INFO [RS:0;jenkins-hbase4:44631] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 19:41:44,092 INFO [RS:0;jenkins-hbase4:44631] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 19:41:44,092 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 19:41:44,093 INFO [RS:0;jenkins-hbase4:44631] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 19:41:44,093 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 9869f769c7045fb07f1013a482eb9f83 2023-07-12 19:41:44,093 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 0c1b9295d50da7c1350cece1d5853d2c 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 9014d82a1855eaedeb89f1edd3e032ac 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 3c735bb63690fff52349b93445eb2aa2 2023-07-12 19:41:44,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9869f769c7045fb07f1013a482eb9f83, disabling compactions & flushes 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for cabde14ae3b55e4b2cc7f9e1b1f244b0 2023-07-12 19:41:44,094 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 2eb66a3eec48e0be370ce75cea0caeae 2023-07-12 19:41:44,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 05380a24dc8e79f5b99fe9c98393c209 2023-07-12 19:41:44,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. after waiting 0 ms 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for c52d439f063558619e98c674c204fb3f 2023-07-12 19:41:44,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for ba20086e056a19e754e00b6bd66152a9 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 997e5e40c84648e410710e2c371a54e7 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for b7f092ed0d4ff500eb86950c457f0615 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 4f6eda388e14a12df47ea1363fceed07 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 3ed8dd7a6148cfa9771a9de1748ddf55 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for af83dc5342faa5fe20a6e4dc7a578aaa 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:44,094 INFO [RS:0;jenkins-hbase4:44631] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 19:41:44,094 DEBUG [RS:0;jenkins-hbase4:44631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x131ff9bf to 127.0.0.1:57973 2023-07-12 19:41:44,094 DEBUG [RS:0;jenkins-hbase4:44631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:44,095 INFO [RS:0;jenkins-hbase4:44631] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 19:41:44,095 INFO [RS:0;jenkins-hbase4:44631] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 19:41:44,095 INFO [RS:0;jenkins-hbase4:44631] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 19:41:44,095 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 19:41:44,095 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1474): Waiting on 16 regions to close 2023-07-12 19:41:44,095 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:41:44,096 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:41:44,096 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:41:44,096 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:41:44,096 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:41:44,097 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.96 KB heapSize=34.40 KB 2023-07-12 19:41:44,098 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1504): Waiting on 05380a24dc8e79f5b99fe9c98393c209, 0c1b9295d50da7c1350cece1d5853d2c, 1588230740, 2eb66a3eec48e0be370ce75cea0caeae, 3c735bb63690fff52349b93445eb2aa2, 3ed8dd7a6148cfa9771a9de1748ddf55, 4f6eda388e14a12df47ea1363fceed07, 9014d82a1855eaedeb89f1edd3e032ac, 9869f769c7045fb07f1013a482eb9f83, 997e5e40c84648e410710e2c371a54e7, af83dc5342faa5fe20a6e4dc7a578aaa, b7f092ed0d4ff500eb86950c457f0615, ba20086e056a19e754e00b6bd66152a9, c52d439f063558619e98c674c204fb3f, cabde14ae3b55e4b2cc7f9e1b1f244b0, de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:44,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/9869f769c7045fb07f1013a482eb9f83/recovered.edits/53.seqid, newMaxSeqId=53, maxSeqId=1 2023-07-12 19:41:44,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:41:44,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9869f769c7045fb07f1013a482eb9f83: 2023-07-12 19:41:44,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotSize3,6,1689190833196.9869f769c7045fb07f1013a482eb9f83. 2023-07-12 19:41:44,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0c1b9295d50da7c1350cece1d5853d2c, disabling compactions & flushes 2023-07-12 19:41:44,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:41:44,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:41:44,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. after waiting 0 ms 2023-07-12 19:41:44,101 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:41:44,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/0c1b9295d50da7c1350cece1d5853d2c/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:41:44,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:41:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0c1b9295d50da7c1350cece1d5853d2c: 2023-07-12 19:41:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotSize3,,1689190833196.0c1b9295d50da7c1350cece1d5853d2c. 2023-07-12 19:41:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9014d82a1855eaedeb89f1edd3e032ac, disabling compactions & flushes 2023-07-12 19:41:44,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:41:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:41:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. after waiting 0 ms 2023-07-12 19:41:44,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:41:44,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables2/9014d82a1855eaedeb89f1edd3e032ac/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:41:44,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:41:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9014d82a1855eaedeb89f1edd3e032ac: 2023-07-12 19:41:44,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotsFromTables2,,1689190820273.9014d82a1855eaedeb89f1edd3e032ac. 2023-07-12 19:41:44,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3c735bb63690fff52349b93445eb2aa2, disabling compactions & flushes 2023-07-12 19:41:44,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:41:44,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:41:44,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. after waiting 0 ms 2023-07-12 19:41:44,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:41:44,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces4/3c735bb63690fff52349b93445eb2aa2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:41:44,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:41:44,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3c735bb63690fff52349b93445eb2aa2: 2023-07-12 19:41:44,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed snapshots_from_namespaces:testSnapshotsFromNamespaces4,,1689190843567.3c735bb63690fff52349b93445eb2aa2. 2023-07-12 19:41:44,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cabde14ae3b55e4b2cc7f9e1b1f244b0, disabling compactions & flushes 2023-07-12 19:41:44,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:41:44,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:41:44,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. after waiting 0 ms 2023-07-12 19:41:44,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:41:44,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables0/cabde14ae3b55e4b2cc7f9e1b1f244b0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:41:44,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:41:44,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cabde14ae3b55e4b2cc7f9e1b1f244b0: 2023-07-12 19:41:44,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotsFromTables0,,1689190817014.cabde14ae3b55e4b2cc7f9e1b1f244b0. 2023-07-12 19:41:44,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2eb66a3eec48e0be370ce75cea0caeae, disabling compactions & flushes 2023-07-12 19:41:44,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:41:44,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:41:44,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. after waiting 0 ms 2023-07-12 19:41:44,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:41:44,122 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/2eb66a3eec48e0be370ce75cea0caeae/recovered.edits/53.seqid, newMaxSeqId=53, maxSeqId=1 2023-07-12 19:41:44,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:41:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2eb66a3eec48e0be370ce75cea0caeae: 2023-07-12 19:41:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotSize3,9,1689190833196.2eb66a3eec48e0be370ce75cea0caeae. 2023-07-12 19:41:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 05380a24dc8e79f5b99fe9c98393c209, disabling compactions & flushes 2023-07-12 19:41:44,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:41:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:41:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. after waiting 0 ms 2023-07-12 19:41:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:41:44,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 05380a24dc8e79f5b99fe9c98393c209 2/2 column families, dataSize=14.03 KB heapSize=30.39 KB 2023-07-12 19:41:44,149 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:41:44,298 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1504): Waiting on 05380a24dc8e79f5b99fe9c98393c209, 1588230740, 3ed8dd7a6148cfa9771a9de1748ddf55, 4f6eda388e14a12df47ea1363fceed07, 997e5e40c84648e410710e2c371a54e7, af83dc5342faa5fe20a6e4dc7a578aaa, b7f092ed0d4ff500eb86950c457f0615, ba20086e056a19e754e00b6bd66152a9, c52d439f063558619e98c674c204fb3f, de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:44,498 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1504): Waiting on 05380a24dc8e79f5b99fe9c98393c209, 1588230740, 3ed8dd7a6148cfa9771a9de1748ddf55, 4f6eda388e14a12df47ea1363fceed07, 997e5e40c84648e410710e2c371a54e7, af83dc5342faa5fe20a6e4dc7a578aaa, b7f092ed0d4ff500eb86950c457f0615, ba20086e056a19e754e00b6bd66152a9, c52d439f063558619e98c674c204fb3f, de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:44,507 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=19.68 KB at sequenceid=67 (bloomFilter=false), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/.tmp/info/9a45535df3ec429aa9b3b2923bb549f7 2023-07-12 19:41:44,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=748 B at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/.tmp/q/d209c65d34ad459dab50300c022f2cdf 2023-07-12 19:41:44,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d209c65d34ad459dab50300c022f2cdf 2023-07-12 19:41:44,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.30 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/.tmp/u/42f51e2ee5ee413bbb2ada7302484cc9 2023-07-12 19:41:44,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 42f51e2ee5ee413bbb2ada7302484cc9 2023-07-12 19:41:44,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/.tmp/q/d209c65d34ad459dab50300c022f2cdf as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/q/d209c65d34ad459dab50300c022f2cdf 2023-07-12 19:41:44,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d209c65d34ad459dab50300c022f2cdf 2023-07-12 19:41:44,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/q/d209c65d34ad459dab50300c022f2cdf, entries=8, sequenceid=184, filesize=5.2 K 2023-07-12 19:41:44,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/.tmp/u/42f51e2ee5ee413bbb2ada7302484cc9 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/u/42f51e2ee5ee413bbb2ada7302484cc9 2023-07-12 19:41:44,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 42f51e2ee5ee413bbb2ada7302484cc9 2023-07-12 19:41:44,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/u/42f51e2ee5ee413bbb2ada7302484cc9, entries=15, sequenceid=184, filesize=5.8 K 2023-07-12 19:41:44,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~14.03 KB/14367, heapSize ~30.36 KB/31088, currentSize=0 B/0 for 05380a24dc8e79f5b99fe9c98393c209 in 430ms, sequenceid=184, compaction requested=false 2023-07-12 19:41:44,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/quota/05380a24dc8e79f5b99fe9c98393c209/recovered.edits/187.seqid, newMaxSeqId=187, maxSeqId=1 2023-07-12 19:41:44,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:41:44,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 05380a24dc8e79f5b99fe9c98393c209: 2023-07-12 19:41:44,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:quota,,1689190815891.05380a24dc8e79f5b99fe9c98393c209. 2023-07-12 19:41:44,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c52d439f063558619e98c674c204fb3f, disabling compactions & flushes 2023-07-12 19:41:44,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:41:44,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:41:44,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. after waiting 0 ms 2023-07-12 19:41:44,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:41:44,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/snapshots_from_namespaces/testSnapshotsFromNamespaces5/c52d439f063558619e98c674c204fb3f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:41:44,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:41:44,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c52d439f063558619e98c674c204fb3f: 2023-07-12 19:41:44,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed snapshots_from_namespaces:testSnapshotsFromNamespaces5,,1689190844179.c52d439f063558619e98c674c204fb3f. 2023-07-12 19:41:44,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ba20086e056a19e754e00b6bd66152a9, disabling compactions & flushes 2023-07-12 19:41:44,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:41:44,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:41:44,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. after waiting 0 ms 2023-07-12 19:41:44,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:41:44,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromNamespaces6/ba20086e056a19e754e00b6bd66152a9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:41:44,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:41:44,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ba20086e056a19e754e00b6bd66152a9: 2023-07-12 19:41:44,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotsFromNamespaces6,,1689190845291.ba20086e056a19e754e00b6bd66152a9. 2023-07-12 19:41:44,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 997e5e40c84648e410710e2c371a54e7, disabling compactions & flushes 2023-07-12 19:41:44,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:41:44,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:41:44,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. after waiting 0 ms 2023-07-12 19:41:44,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:41:44,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 997e5e40c84648e410710e2c371a54e7 1/1 column families, dataSize=155 B heapSize=640 B 2023-07-12 19:41:44,698 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1504): Waiting on 1588230740, 3ed8dd7a6148cfa9771a9de1748ddf55, 4f6eda388e14a12df47ea1363fceed07, 997e5e40c84648e410710e2c371a54e7, af83dc5342faa5fe20a6e4dc7a578aaa, b7f092ed0d4ff500eb86950c457f0615, de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:44,899 DEBUG [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1504): Waiting on 1588230740, 3ed8dd7a6148cfa9771a9de1748ddf55, 4f6eda388e14a12df47ea1363fceed07, 997e5e40c84648e410710e2c371a54e7, af83dc5342faa5fe20a6e4dc7a578aaa, b7f092ed0d4ff500eb86950c457f0615, de854fce7414330f8a87a4432783eca1 2023-07-12 19:41:44,908 DEBUG [master/jenkins-hbase4:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2f2592bf to 127.0.0.1:57973 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:41:44,911 DEBUG [master/jenkins-hbase4:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@307e2fb8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:41:44,912 DEBUG [hconnection-0x432c287-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:41:44,913 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:41:44,913 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=44631] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase4.apache.org,44631,1689190811374 stopping at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1527) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:41:44,914 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=44631] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 131 connection: 172.31.14.131:35376 deadline: 1689190964913, exception=org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase4.apache.org,44631,1689190811374 stopping 2023-07-12 19:41:44,918 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.28 KB at sequenceid=67 (bloomFilter=false), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/.tmp/table/7bb15bd7446d4f1cbf7a0cb140b05a6f 2023-07-12 19:41:44,922 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/.tmp/info/9a45535df3ec429aa9b3b2923bb549f7 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/info/9a45535df3ec429aa9b3b2923bb549f7 2023-07-12 19:41:44,925 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/info/9a45535df3ec429aa9b3b2923bb549f7, entries=150, sequenceid=67, filesize=24.8 K 2023-07-12 19:41:44,926 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/.tmp/table/7bb15bd7446d4f1cbf7a0cb140b05a6f as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/table/7bb15bd7446d4f1cbf7a0cb140b05a6f 2023-07-12 19:41:44,930 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/table/7bb15bd7446d4f1cbf7a0cb140b05a6f, entries=22, sequenceid=67, filesize=5.9 K 2023-07-12 19:41:44,931 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.96 KB/21461, heapSize ~34.12 KB/34936, currentSize=0 B/0 for 1588230740 in 835ms, sequenceid=67, compaction requested=false 2023-07-12 19:41:44,938 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/meta/1588230740/recovered.edits/70.seqid, newMaxSeqId=70, maxSeqId=1 2023-07-12 19:41:44,938 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:41:44,939 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:41:44,939 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:41:44,939 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 19:41:44,949 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 19:41:44,949 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 19:41:44,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=155 B at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7/.tmp/info/200e1968534047d7a06aae6126a41179 2023-07-12 19:41:44,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7/.tmp/info/200e1968534047d7a06aae6126a41179 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7/info/200e1968534047d7a06aae6126a41179 2023-07-12 19:41:44,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7/info/200e1968534047d7a06aae6126a41179, entries=3, sequenceid=7, filesize=4.9 K 2023-07-12 19:41:44,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~155 B/155, heapSize ~624 B/624, currentSize=0 B/0 for 997e5e40c84648e410710e2c371a54e7 in 416ms, sequenceid=7, compaction requested=false 2023-07-12 19:41:44,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/hbase/namespace/997e5e40c84648e410710e2c371a54e7/recovered.edits/10.seqid, newMaxSeqId=10, maxSeqId=1 2023-07-12 19:41:44,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:41:44,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 997e5e40c84648e410710e2c371a54e7: 2023-07-12 19:41:44,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689190814350.997e5e40c84648e410710e2c371a54e7. 2023-07-12 19:41:44,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b7f092ed0d4ff500eb86950c457f0615, disabling compactions & flushes 2023-07-12 19:41:44,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:41:44,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:41:44,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. after waiting 0 ms 2023-07-12 19:41:44,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:41:44,989 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: SpaceQuotaRefresherChore was stopped 2023-07-12 19:41:44,989 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: RegionSizeReportingChore was stopped 2023-07-12 19:41:44,990 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: FileSystemUtilizationChore was stopped 2023-07-12 19:41:44,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/b7f092ed0d4ff500eb86950c457f0615/recovered.edits/55.seqid, newMaxSeqId=55, maxSeqId=1 2023-07-12 19:41:44,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:41:44,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b7f092ed0d4ff500eb86950c457f0615: 2023-07-12 19:41:44,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotSize3,0,1689190833196.b7f092ed0d4ff500eb86950c457f0615. 2023-07-12 19:41:44,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4f6eda388e14a12df47ea1363fceed07, disabling compactions & flushes 2023-07-12 19:41:44,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:41:44,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:41:44,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. after waiting 0 ms 2023-07-12 19:41:44,993 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:41:44,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotSize3/4f6eda388e14a12df47ea1363fceed07/recovered.edits/55.seqid, newMaxSeqId=55, maxSeqId=1 2023-07-12 19:41:44,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:41:44,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4f6eda388e14a12df47ea1363fceed07: 2023-07-12 19:41:44,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotSize3,3,1689190833196.4f6eda388e14a12df47ea1363fceed07. 2023-07-12 19:41:44,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing de854fce7414330f8a87a4432783eca1, disabling compactions & flushes 2023-07-12 19:41:44,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:44,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:44,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. after waiting 0 ms 2023-07-12 19:41:44,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:45,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testRemovedSnapshots8/de854fce7414330f8a87a4432783eca1/recovered.edits/29.seqid, newMaxSeqId=29, maxSeqId=1 2023-07-12 19:41:45,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:45,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for de854fce7414330f8a87a4432783eca1: 2023-07-12 19:41:45,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testRemovedSnapshots8,,1689190888151.de854fce7414330f8a87a4432783eca1. 2023-07-12 19:41:45,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3ed8dd7a6148cfa9771a9de1748ddf55, disabling compactions & flushes 2023-07-12 19:41:45,003 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:41:45,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:41:45,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. after waiting 0 ms 2023-07-12 19:41:45,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:41:45,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testSnapshotsFromTables1/3ed8dd7a6148cfa9771a9de1748ddf55/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:41:45,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:41:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3ed8dd7a6148cfa9771a9de1748ddf55: 2023-07-12 19:41:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testSnapshotsFromTables1,,1689190819155.3ed8dd7a6148cfa9771a9de1748ddf55. 2023-07-12 19:41:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing af83dc5342faa5fe20a6e4dc7a578aaa, disabling compactions & flushes 2023-07-12 19:41:45,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. after waiting 0 ms 2023-07-12 19:41:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:45,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/data/default/testBucketingFilesToSnapshots7/af83dc5342faa5fe20a6e4dc7a578aaa/recovered.edits/31.seqid, newMaxSeqId=31, maxSeqId=1 2023-07-12 19:41:45,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:45,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for af83dc5342faa5fe20a6e4dc7a578aaa: 2023-07-12 19:41:45,010 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testBucketingFilesToSnapshots7,,1689190856698.af83dc5342faa5fe20a6e4dc7a578aaa. 2023-07-12 19:41:45,018 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44631] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase4.apache.org,44631,1689190811374 stopping at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1527) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:41:45,019 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=44631] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 131 connection: 172.31.14.131:35376 deadline: 1689190964912, exception=org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase4.apache.org,44631,1689190811374 stopping 2023-07-12 19:41:45,099 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,44631,1689190811374; all regions closed. 2023-07-12 19:41:45,099 DEBUG [RS:0;jenkins-hbase4:44631] quotas.QuotaCache(100): Stopping QuotaRefresherChore chore. 2023-07-12 19:41:45,107 DEBUG [RS:0;jenkins-hbase4:44631] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/oldWALs 2023-07-12 19:41:45,107 INFO [RS:0;jenkins-hbase4:44631] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44631%2C1689190811374.meta:.meta(num 1689190814093) 2023-07-12 19:41:45,110 DEBUG [RS:0;jenkins-hbase4:44631] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/oldWALs 2023-07-12 19:41:45,111 INFO [RS:0;jenkins-hbase4:44631] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44631%2C1689190811374:(num 1689190813107) 2023-07-12 19:41:45,111 DEBUG [RS:0;jenkins-hbase4:44631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,111 INFO [RS:0;jenkins-hbase4:44631] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:41:45,111 INFO [RS:0;jenkins-hbase4:44631] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=15000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 19:41:45,111 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:41:45,112 INFO [RS:0;jenkins-hbase4:44631] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44631 2023-07-12 19:41:45,120 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44631,1689190811374 2023-07-12 19:41:45,120 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:41:45,121 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:41:45,122 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44631,1689190811374] 2023-07-12 19:41:45,122 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44631,1689190811374; numProcessing=1 2023-07-12 19:41:45,123 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44631,1689190811374 already deleted, retry=false 2023-07-12 19:41:45,123 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,44631,1689190811374 expired; onlineServers=0 2023-07-12 19:41:45,124 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,33647,1689190810081' ***** 2023-07-12 19:41:45,124 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 19:41:45,124 DEBUG [M:0;jenkins-hbase4:33647] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30e310a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:41:45,124 INFO [M:0;jenkins-hbase4:33647] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,33647,1689190810081 2023-07-12 19:41:45,124 INFO [M:0;jenkins-hbase4:33647] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,33647,1689190810081; all regions closed. 2023-07-12 19:41:45,124 DEBUG [M:0;jenkins-hbase4:33647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,124 DEBUG [M:0;jenkins-hbase4:33647] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:41:45,124 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:41:45,124 DEBUG [M:0;jenkins-hbase4:33647] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:41:45,124 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689190812826] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689190812826,5,FailOnTimeoutGroup] 2023-07-12 19:41:45,124 INFO [M:0;jenkins-hbase4:33647] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:41:45,124 INFO [M:0;jenkins-hbase4:33647] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:41:45,124 WARN [master/jenkins-hbase4:0.Chore.1] quotas.QuotaRetriever$Iter(131): java.io.InterruptedIOException at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:220) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:266) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:309) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:594) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:989) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:866) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:157) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:266) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:309) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:594) at org.apache.hadoop.hbase.quotas.QuotaRetriever.next(QuotaRetriever.java:103) at org.apache.hadoop.hbase.quotas.QuotaRetriever$Iter.(QuotaRetriever.java:129) at org.apache.hadoop.hbase.quotas.QuotaRetriever.iterator(QuotaRetriever.java:121) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsToComputeSize(SnapshotQuotaObserverChore.java:171) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:106) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:41:45,125 INFO [M:0;jenkins-hbase4:33647] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [ScheduledChore name=QuotaObserverChore, period=1000, unit=MILLISECONDS] on shutdown 2023-07-12 19:41:45,124 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689190812827] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689190812827,5,FailOnTimeoutGroup] 2023-07-12 19:41:45,125 DEBUG [M:0;jenkins-hbase4:33647] master.HMaster(1512): Stopping service threads 2023-07-12 19:41:45,125 WARN [master/jenkins-hbase4:0.Chore.1] quotas.SnapshotQuotaObserverChore(98): Failed to compute the size of snapshots, will retry org.apache.hadoop.hbase.exceptions.MasterStoppedException at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2843) at org.apache.hadoop.hbase.master.MasterRpcServices.getCompletedSnapshots(MasterRpcServices.java:1002) at org.apache.hadoop.hbase.client.ShortCircuitMasterConnection.getCompletedSnapshots(ShortCircuitMasterConnection.java:496) at org.apache.hadoop.hbase.client.HBaseAdmin$70.rpcCall(HBaseAdmin.java:2793) at org.apache.hadoop.hbase.client.HBaseAdmin$70.rpcCall(HBaseAdmin.java:2789) at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:99) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2946) at org.apache.hadoop.hbase.client.HBaseAdmin.listSnapshots(HBaseAdmin.java:2788) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsFromTables(SnapshotQuotaObserverChore.java:199) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsToComputeSize(SnapshotQuotaObserverChore.java:188) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:106) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:41:45,125 INFO [M:0;jenkins-hbase4:33647] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:41:45,126 ERROR [M:0;jenkins-hbase4:33647] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-9,5,PEWorkerGroup] 2023-07-12 19:41:45,126 INFO [M:0;jenkins-hbase4:33647] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:41:45,126 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:41:45,131 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:41:45,131 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:41:45,131 DEBUG [M:0;jenkins-hbase4:33647] zookeeper.ZKUtil(398): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:41:45,132 WARN [M:0;jenkins-hbase4:33647] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:41:45,132 INFO [M:0;jenkins-hbase4:33647] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:41:45,132 INFO [M:0;jenkins-hbase4:33647] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 19:41:45,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:41:45,132 DEBUG [M:0;jenkins-hbase4:33647] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:41:45,132 INFO [M:0;jenkins-hbase4:33647] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:41:45,132 DEBUG [M:0;jenkins-hbase4:33647] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:41:45,132 DEBUG [M:0;jenkins-hbase4:33647] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:41:45,132 DEBUG [M:0;jenkins-hbase4:33647] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:41:45,132 INFO [M:0;jenkins-hbase4:33647] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=236.83 KB heapSize=282.53 KB 2023-07-12 19:41:45,222 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:41:45,222 INFO [RS:0;jenkins-hbase4:44631] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,44631,1689190811374; zookeeper connection closed. 2023-07-12 19:41:45,222 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): regionserver:44631-0x1015b0cc17f0001, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:41:45,223 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@41736e43] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@41736e43 2023-07-12 19:41:45,223 INFO [Listener at localhost/32953] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-07-12 19:41:45,542 INFO [M:0;jenkins-hbase4:33647] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=236.83 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9f827c1f22c8471c9709031d900a7be2 2023-07-12 19:41:45,558 INFO [M:0;jenkins-hbase4:33647] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9f827c1f22c8471c9709031d900a7be2 2023-07-12 19:41:45,559 DEBUG [M:0;jenkins-hbase4:33647] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9f827c1f22c8471c9709031d900a7be2 as hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9f827c1f22c8471c9709031d900a7be2 2023-07-12 19:41:45,576 INFO [M:0;jenkins-hbase4:33647] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9f827c1f22c8471c9709031d900a7be2 2023-07-12 19:41:45,576 INFO [M:0;jenkins-hbase4:33647] regionserver.HStore(1080): Added hdfs://localhost:35215/user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9f827c1f22c8471c9709031d900a7be2, entries=75, sequenceid=553, filesize=22.4 K 2023-07-12 19:41:45,577 INFO [M:0;jenkins-hbase4:33647] regionserver.HRegion(2948): Finished flush of dataSize ~236.83 KB/242515, heapSize ~282.52 KB/289296, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 445ms, sequenceid=553, compaction requested=false 2023-07-12 19:41:45,578 INFO [M:0;jenkins-hbase4:33647] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:41:45,578 DEBUG [M:0;jenkins-hbase4:33647] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:41:45,580 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/43f85b11-a02a-31e5-472d-bb1dc15d6151/MasterData/WALs/jenkins-hbase4.apache.org,33647,1689190810081/jenkins-hbase4.apache.org%2C33647%2C1689190810081.1689190812205 not finished, retry = 0 2023-07-12 19:41:45,681 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:41:45,681 INFO [M:0;jenkins-hbase4:33647] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:41:45,682 INFO [M:0;jenkins-hbase4:33647] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33647 2023-07-12 19:41:45,684 DEBUG [M:0;jenkins-hbase4:33647] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,33647,1689190810081 already deleted, retry=false 2023-07-12 19:41:45,786 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:41:45,786 INFO [M:0;jenkins-hbase4:33647] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,33647,1689190810081; zookeeper connection closed. 2023-07-12 19:41:45,786 DEBUG [Listener at localhost/32953-EventThread] zookeeper.ZKWatcher(600): master:33647-0x1015b0cc17f0000, quorum=127.0.0.1:57973, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:41:45,788 WARN [Listener at localhost/32953] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 19:41:45,805 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f2592bf to 127.0.0.1:57973 2023-07-12 19:41:45,805 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,806 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a506f0b to 127.0.0.1:57973 2023-07-12 19:41:45,806 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,806 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b8ff6db to 127.0.0.1:57973 2023-07-12 19:41:45,806 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,806 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63e7813b to 127.0.0.1:57973 2023-07-12 19:41:45,806 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,806 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c523f9d to 127.0.0.1:57973 2023-07-12 19:41:45,806 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,806 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5808dfd7 to 127.0.0.1:57973 2023-07-12 19:41:45,806 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,806 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79cf84e1 to 127.0.0.1:57973 2023-07-12 19:41:45,806 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,807 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41437606 to 127.0.0.1:57973 2023-07-12 19:41:45,807 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,807 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ceff890 to 127.0.0.1:57973 2023-07-12 19:41:45,807 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,807 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ff733e to 127.0.0.1:57973 2023-07-12 19:41:45,807 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,807 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bf0df20 to 127.0.0.1:57973 2023-07-12 19:41:45,807 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,807 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14f9cac1 to 127.0.0.1:57973 2023-07-12 19:41:45,807 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,807 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20457466 to 127.0.0.1:57973 2023-07-12 19:41:45,807 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,807 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54ffa64f to 127.0.0.1:57973 2023-07-12 19:41:45,807 INFO [Listener at localhost/32953] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:41:45,807 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,807 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x374fea4c to 127.0.0.1:57973 2023-07-12 19:41:45,807 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,808 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4dbe8b35 to 127.0.0.1:57973 2023-07-12 19:41:45,808 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:41:45,912 WARN [BP-18444718-172.31.14.131-1689190807059 heartbeating to localhost/127.0.0.1:35215] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 19:41:45,912 WARN [BP-18444718-172.31.14.131-1689190807059 heartbeating to localhost/127.0.0.1:35215] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-18444718-172.31.14.131-1689190807059 (Datanode Uuid 16e71f0f-a9d5-410c-b866-01f5fae0b1cd) service to localhost/127.0.0.1:35215 2023-07-12 19:41:45,914 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/cluster_cb8e40ba-4554-a1e8-3e84-9973d51e9321/dfs/data/data1/current/BP-18444718-172.31.14.131-1689190807059] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:41:45,914 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/0b8ce3e6-979e-2ad1-32e9-088a663b49be/cluster_cb8e40ba-4554-a1e8-3e84-9973d51e9321/dfs/data/data2/current/BP-18444718-172.31.14.131-1689190807059] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:41:45,947 INFO [Listener at localhost/32953] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:41:46,063 INFO [Listener at localhost/32953] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 19:41:46,250 INFO [Listener at localhost/32953] hbase.HBaseTestingUtility(1293): Minicluster is down