2023-07-12 15:39:35,806 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.quotas.TestSpaceQuotasWithSnapshots timeout: 13 mins 2023-07-12 15:39:36,056 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff 2023-07-12 15:39:36,072 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 15:39:36,073 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/cluster_25c97bd4-4232-2ad9-6328-0c863aed8e02, deleteOnExit=true 2023-07-12 15:39:36,073 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 15:39:36,073 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/test.cache.data in system properties and HBase conf 2023-07-12 15:39:36,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 15:39:36,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/hadoop.log.dir in system properties and HBase conf 2023-07-12 15:39:36,075 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 15:39:36,075 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 15:39:36,075 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 15:39:36,191 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 15:39:36,578 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 15:39:36,582 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:39:36,583 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 15:39:36,583 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 15:39:36,583 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:39:36,584 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 15:39:36,584 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 15:39:36,584 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 15:39:36,585 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:39:36,585 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 15:39:36,586 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/nfs.dump.dir in system properties and HBase conf 2023-07-12 15:39:36,586 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/java.io.tmpdir in system properties and HBase conf 2023-07-12 15:39:36,586 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 15:39:36,587 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 15:39:36,587 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 15:39:37,141 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:39:37,145 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:39:37,440 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 15:39:37,604 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 15:39:37,619 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:39:37,659 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:39:37,693 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/java.io.tmpdir/Jetty_localhost_44771_hdfs____5il94r/webapp 2023-07-12 15:39:37,841 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44771 2023-07-12 15:39:37,858 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 15:39:37,858 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 15:39:38,361 WARN [Listener at localhost/38173] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:39:38,421 WARN [Listener at localhost/38173] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 15:39:38,438 WARN [Listener at localhost/38173] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 15:39:38,444 INFO [Listener at localhost/38173] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 15:39:38,448 INFO [Listener at localhost/38173] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/java.io.tmpdir/Jetty_localhost_37807_datanode____q4560y/webapp 2023-07-12 15:39:38,542 INFO [Listener at localhost/38173] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37807 2023-07-12 15:39:38,832 WARN [Listener at localhost/38241] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 15:39:39,245 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x50eb00e17b2360cb: Processing first storage report for DS-81fd12ba-0a3e-4a96-814c-a95c78be509b from datanode a2d3ebee-8da6-4387-bf6f-5ce1660e5ee1 2023-07-12 15:39:39,247 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x50eb00e17b2360cb: from storage DS-81fd12ba-0a3e-4a96-814c-a95c78be509b node DatanodeRegistration(127.0.0.1:40285, datanodeUuid=a2d3ebee-8da6-4387-bf6f-5ce1660e5ee1, infoPort=41281, infoSecurePort=0, ipcPort=38241, storageInfo=lv=-57;cid=testClusterID;nsid=1253912376;c=1689176377226), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 15:39:39,247 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x50eb00e17b2360cb: Processing first storage report for DS-f562af6f-57d8-4a54-9f46-d1bf025260ad from datanode a2d3ebee-8da6-4387-bf6f-5ce1660e5ee1 2023-07-12 15:39:39,247 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x50eb00e17b2360cb: from storage DS-f562af6f-57d8-4a54-9f46-d1bf025260ad node DatanodeRegistration(127.0.0.1:40285, datanodeUuid=a2d3ebee-8da6-4387-bf6f-5ce1660e5ee1, infoPort=41281, infoSecurePort=0, ipcPort=38241, storageInfo=lv=-57;cid=testClusterID;nsid=1253912376;c=1689176377226), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 15:39:39,302 DEBUG [Listener at localhost/38241] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff 2023-07-12 15:39:39,371 INFO [Listener at localhost/38241] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/cluster_25c97bd4-4232-2ad9-6328-0c863aed8e02/zookeeper_0, clientPort=51623, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/cluster_25c97bd4-4232-2ad9-6328-0c863aed8e02/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/cluster_25c97bd4-4232-2ad9-6328-0c863aed8e02/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 15:39:39,387 INFO [Listener at localhost/38241] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51623 2023-07-12 15:39:39,397 INFO [Listener at localhost/38241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:39:39,399 INFO [Listener at localhost/38241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:39:40,049 INFO [Listener at localhost/38241] util.FSUtils(471): Created version file at hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788 with version=8 2023-07-12 15:39:40,050 INFO [Listener at localhost/38241] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/hbase-staging 2023-07-12 15:39:40,371 INFO [Listener at localhost/38241] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 15:39:40,837 INFO [Listener at localhost/38241] client.ConnectionUtils(127): master/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:39:40,869 INFO [Listener at localhost/38241] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:39:40,870 INFO [Listener at localhost/38241] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:39:40,870 INFO [Listener at localhost/38241] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:39:40,870 INFO [Listener at localhost/38241] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:39:40,870 INFO [Listener at localhost/38241] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:39:41,010 INFO [Listener at localhost/38241] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:39:41,083 DEBUG [Listener at localhost/38241] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 15:39:41,176 INFO [Listener at localhost/38241] ipc.NettyRpcServer(120): Bind to /172.31.10.131:37087 2023-07-12 15:39:41,186 INFO [Listener at localhost/38241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:39:41,188 INFO [Listener at localhost/38241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:39:41,208 INFO [Listener at localhost/38241] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37087 connecting to ZooKeeper ensemble=127.0.0.1:51623 2023-07-12 15:39:41,250 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:370870x0, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:39:41,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:37087-0x101c5ccda360000 connected 2023-07-12 15:39:41,275 DEBUG [Listener at localhost/38241] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:39:41,276 DEBUG [Listener at localhost/38241] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:39:41,279 DEBUG [Listener at localhost/38241] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:39:41,290 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37087 2023-07-12 15:39:41,291 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37087 2023-07-12 15:39:41,292 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37087 2023-07-12 15:39:41,292 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37087 2023-07-12 15:39:41,292 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37087 2023-07-12 15:39:41,298 INFO [Listener at localhost/38241] master.HMaster(444): hbase.rootdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788, hbase.cluster.distributed=false 2023-07-12 15:39:41,370 INFO [Listener at localhost/38241] client.ConnectionUtils(127): regionserver/jenkins-hbase5:0 server-side Connection retries=45 2023-07-12 15:39:41,370 INFO [Listener at localhost/38241] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:39:41,370 INFO [Listener at localhost/38241] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 15:39:41,370 INFO [Listener at localhost/38241] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 15:39:41,371 INFO [Listener at localhost/38241] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 15:39:41,371 INFO [Listener at localhost/38241] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 15:39:41,377 INFO [Listener at localhost/38241] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 15:39:41,380 INFO [Listener at localhost/38241] ipc.NettyRpcServer(120): Bind to /172.31.10.131:45089 2023-07-12 15:39:41,383 INFO [Listener at localhost/38241] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 15:39:41,415 DEBUG [Listener at localhost/38241] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 15:39:41,416 INFO [Listener at localhost/38241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:39:41,420 INFO [Listener at localhost/38241] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:39:41,422 INFO [Listener at localhost/38241] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45089 connecting to ZooKeeper ensemble=127.0.0.1:51623 2023-07-12 15:39:41,435 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:450890x0, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 15:39:41,443 DEBUG [Listener at localhost/38241] zookeeper.ZKUtil(164): regionserver:450890x0, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:39:41,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:45089-0x101c5ccda360001 connected 2023-07-12 15:39:41,446 DEBUG [Listener at localhost/38241] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:39:41,447 DEBUG [Listener at localhost/38241] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 15:39:41,448 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45089 2023-07-12 15:39:41,449 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45089 2023-07-12 15:39:41,455 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45089 2023-07-12 15:39:41,459 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45089 2023-07-12 15:39:41,463 DEBUG [Listener at localhost/38241] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45089 2023-07-12 15:39:41,465 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:39:41,475 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:39:41,476 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:39:41,495 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:39:41,495 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 15:39:41,495 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:39:41,496 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:39:41,498 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase5.apache.org,37087,1689176380200 from backup master directory 2023-07-12 15:39:41,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 15:39:41,501 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:39:41,501 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 15:39:41,502 WARN [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:39:41,502 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:39:41,505 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 15:39:41,506 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 15:39:41,587 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/hbase.id with ID: 60dc30fa-b133-40e7-a049-5b3d0cfcdb3b 2023-07-12 15:39:41,631 INFO [master/jenkins-hbase5:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 15:39:41,645 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:39:41,683 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2e245caf to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:41,720 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4300fcd5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:41,742 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:39:41,744 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 15:39:41,761 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 15:39:41,762 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 15:39:41,763 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:39:41,767 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:39:41,768 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:39:41,800 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store-tmp 2023-07-12 15:39:41,825 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:41,826 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:39:41,826 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:39:41,826 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:39:41,826 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:39:41,826 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:39:41,826 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:39:41,826 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:39:41,827 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/WALs/jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:39:41,850 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C37087%2C1689176380200, suffix=, logDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/WALs/jenkins-hbase5.apache.org,37087,1689176380200, archiveDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/oldWALs, maxLogs=10 2023-07-12 15:39:41,901 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40285,DS-81fd12ba-0a3e-4a96-814c-a95c78be509b,DISK] 2023-07-12 15:39:41,909 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:39:41,973 INFO [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/WALs/jenkins-hbase5.apache.org,37087,1689176380200/jenkins-hbase5.apache.org%2C37087%2C1689176380200.1689176381861 2023-07-12 15:39:41,974 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40285,DS-81fd12ba-0a3e-4a96-814c-a95c78be509b,DISK]] 2023-07-12 15:39:41,975 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:39:41,975 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:41,979 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:39:41,980 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:39:42,041 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:39:42,049 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 15:39:42,075 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 15:39:42,088 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:42,094 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:39:42,096 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:39:42,113 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 15:39:42,124 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:39:42,125 INFO [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70259806, jitterRate=0.04695269465446472}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:39:42,125 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:39:42,127 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 15:39:42,150 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-12 15:39:42,150 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-12 15:39:42,153 INFO [master/jenkins-hbase5:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 15:39:42,155 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 15:39:42,191 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 35 msec 2023-07-12 15:39:42,191 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 15:39:42,221 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 15:39:42,227 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 15:39:42,258 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 15:39:42,264 INFO [master/jenkins-hbase5:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 15:39:42,266 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 15:39:42,273 INFO [master/jenkins-hbase5:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 15:39:42,278 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 15:39:42,281 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:39:42,283 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 15:39:42,284 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 15:39:42,296 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 15:39:42,301 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:39:42,301 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 15:39:42,301 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:39:42,302 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase5.apache.org,37087,1689176380200, sessionid=0x101c5ccda360000, setting cluster-up flag (Was=false) 2023-07-12 15:39:42,322 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:39:42,327 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 15:39:42,329 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:39:42,335 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:39:42,342 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 15:39:42,344 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:39:42,347 WARN [master/jenkins-hbase5:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp 2023-07-12 15:39:42,368 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(951): ClusterId : 60dc30fa-b133-40e7-a049-5b3d0cfcdb3b 2023-07-12 15:39:42,373 DEBUG [RS:0;jenkins-hbase5:45089] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 15:39:42,379 DEBUG [RS:0;jenkins-hbase5:45089] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 15:39:42,380 DEBUG [RS:0;jenkins-hbase5:45089] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 15:39:42,383 DEBUG [RS:0;jenkins-hbase5:45089] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 15:39:42,384 DEBUG [RS:0;jenkins-hbase5:45089] zookeeper.ReadOnlyZKClient(139): Connect 0x3eb8bc3b to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:42,406 INFO [master/jenkins-hbase5:0:becomeActiveMaster] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.quotas.MasterQuotasObserver loaded, priority=536870911. 2023-07-12 15:39:42,412 DEBUG [RS:0;jenkins-hbase5:45089] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@307f459c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:42,413 DEBUG [RS:0;jenkins-hbase5:45089] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3440d74b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:39:42,454 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase5:45089 2023-07-12 15:39:42,459 INFO [RS:0;jenkins-hbase5:45089] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 15:39:42,459 INFO [RS:0;jenkins-hbase5:45089] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 15:39:42,459 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 15:39:42,463 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,37087,1689176380200 with isa=jenkins-hbase5.apache.org/172.31.10.131:45089, startcode=1689176381369 2023-07-12 15:39:42,482 DEBUG [RS:0;jenkins-hbase5:45089] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 15:39:42,503 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 15:39:42,531 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:39:42,531 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:39:42,531 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:39:42,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=5, maxPoolSize=5 2023-07-12 15:39:42,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase5:0, corePoolSize=10, maxPoolSize=10 2023-07-12 15:39:42,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:39:42,532 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,543 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689176412543 2023-07-12 15:39:42,545 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 15:39:42,548 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:39:42,548 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 15:39:42,555 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:39:42,555 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43655, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 15:39:42,558 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 15:39:42,567 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 15:39:42,568 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 15:39:42,569 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 15:39:42,569 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 15:39:42,571 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,573 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37087] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:39:42,573 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 15:39:42,575 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 15:39:42,575 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 15:39:42,579 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 15:39:42,580 INFO [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 15:39:42,587 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689176382581,5,FailOnTimeoutGroup] 2023-07-12 15:39:42,587 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689176382587,5,FailOnTimeoutGroup] 2023-07-12 15:39:42,587 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,587 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 15:39:42,589 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,589 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,608 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:39:42,609 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 15:39:42,610 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788 2023-07-12 15:39:42,614 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(2830): Master is not running yet 2023-07-12 15:39:42,614 WARN [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-12 15:39:42,634 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:42,652 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:39:42,656 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/info 2023-07-12 15:39:42,656 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:39:42,658 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:42,658 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:39:42,662 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:39:42,662 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:39:42,663 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:42,664 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:39:42,666 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/table 2023-07-12 15:39:42,667 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:39:42,668 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:42,670 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740 2023-07-12 15:39:42,671 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740 2023-07-12 15:39:42,675 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:39:42,677 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:39:42,681 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:39:42,683 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68363330, jitterRate=0.018693000078201294}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:39:42,683 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:39:42,683 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:39:42,683 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:39:42,683 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:39:42,683 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:39:42,683 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:39:42,684 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:39:42,684 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:39:42,690 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 15:39:42,690 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 15:39:42,703 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 15:39:42,715 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase5.apache.org,37087,1689176380200 with isa=jenkins-hbase5.apache.org/172.31.10.131:45089, startcode=1689176381369 2023-07-12 15:39:42,722 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=37087] master.ServerManager(394): Registering regionserver=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:42,725 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 15:39:42,729 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 15:39:42,731 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788 2023-07-12 15:39:42,731 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:38173 2023-07-12 15:39:42,731 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 15:39:42,736 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:39:42,737 DEBUG [RS:0;jenkins-hbase5:45089] zookeeper.ZKUtil(162): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:42,738 WARN [RS:0;jenkins-hbase5:45089] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 15:39:42,748 INFO [RS:0;jenkins-hbase5:45089] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:39:42,748 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1948): logDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/WALs/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:42,755 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase5.apache.org,45089,1689176381369] 2023-07-12 15:39:42,764 DEBUG [RS:0;jenkins-hbase5:45089] zookeeper.ZKUtil(162): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:42,775 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 15:39:42,784 INFO [RS:0;jenkins-hbase5:45089] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 15:39:42,808 INFO [RS:0;jenkins-hbase5:45089] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 15:39:42,813 INFO [RS:0;jenkins-hbase5:45089] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:39:42,814 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,815 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 15:39:42,825 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=5000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,825 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,826 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,826 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,826 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,827 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,827 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase5:0, corePoolSize=2, maxPoolSize=2 2023-07-12 15:39:42,827 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,827 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,827 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,827 DEBUG [RS:0;jenkins-hbase5:45089] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase5:0, corePoolSize=1, maxPoolSize=1 2023-07-12 15:39:42,839 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,839 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,839 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,839 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=FileSystemUtilizationChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,862 INFO [RS:0;jenkins-hbase5:45089] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 15:39:42,865 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,45089,1689176381369-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,881 DEBUG [jenkins-hbase5:37087] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 15:39:42,884 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,45089,1689176381369, state=OPENING 2023-07-12 15:39:42,885 INFO [RS:0;jenkins-hbase5:45089] regionserver.Replication(203): jenkins-hbase5.apache.org,45089,1689176381369 started 2023-07-12 15:39:42,886 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1637): Serving as jenkins-hbase5.apache.org,45089,1689176381369, RpcServer on jenkins-hbase5.apache.org/172.31.10.131:45089, sessionid=0x101c5ccda360001 2023-07-12 15:39:42,886 DEBUG [RS:0;jenkins-hbase5:45089] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 15:39:42,886 DEBUG [RS:0;jenkins-hbase5:45089] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:42,886 DEBUG [RS:0;jenkins-hbase5:45089] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,45089,1689176381369' 2023-07-12 15:39:42,886 DEBUG [RS:0;jenkins-hbase5:45089] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:39:42,887 DEBUG [RS:0;jenkins-hbase5:45089] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:39:42,887 DEBUG [RS:0;jenkins-hbase5:45089] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 15:39:42,888 DEBUG [RS:0;jenkins-hbase5:45089] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 15:39:42,888 DEBUG [RS:0;jenkins-hbase5:45089] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:42,888 DEBUG [RS:0;jenkins-hbase5:45089] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase5.apache.org,45089,1689176381369' 2023-07-12 15:39:42,888 DEBUG [RS:0;jenkins-hbase5:45089] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:39:42,888 DEBUG [RS:0;jenkins-hbase5:45089] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:39:42,889 DEBUG [RS:0;jenkins-hbase5:45089] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 15:39:42,889 INFO [RS:0;jenkins-hbase5:45089] quotas.RegionServerRpcQuotaManager(67): Initializing RPC quota support 2023-07-12 15:39:42,891 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=QuotaRefresherChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,893 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 15:39:42,893 DEBUG [RS:0;jenkins-hbase5:45089] zookeeper.ZKUtil(398): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/rpc-throttle because node does not exist (not an error) 2023-07-12 15:39:42,894 INFO [RS:0;jenkins-hbase5:45089] quotas.RegionServerRpcQuotaManager(73): Start rpc quota manager and rpc throttle enabled is true 2023-07-12 15:39:42,894 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=SpaceQuotaRefresherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,895 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(166): Chore ScheduledChore name=RegionSizeReportingChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:42,895 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:39:42,896 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:39:42,900 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:39:43,007 INFO [RS:0;jenkins-hbase5:45089] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C45089%2C1689176381369, suffix=, logDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/WALs/jenkins-hbase5.apache.org,45089,1689176381369, archiveDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/oldWALs, maxLogs=32 2023-07-12 15:39:43,025 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40285,DS-81fd12ba-0a3e-4a96-814c-a95c78be509b,DISK] 2023-07-12 15:39:43,028 INFO [RS:0;jenkins-hbase5:45089] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/WALs/jenkins-hbase5.apache.org,45089,1689176381369/jenkins-hbase5.apache.org%2C45089%2C1689176381369.1689176383009 2023-07-12 15:39:43,029 DEBUG [RS:0;jenkins-hbase5:45089] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40285,DS-81fd12ba-0a3e-4a96-814c-a95c78be509b,DISK]] 2023-07-12 15:39:43,084 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:43,087 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 15:39:43,091 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:48164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 15:39:43,102 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 15:39:43,102 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 15:39:43,106 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase5.apache.org%2C45089%2C1689176381369.meta, suffix=.meta, logDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/WALs/jenkins-hbase5.apache.org,45089,1689176381369, archiveDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/oldWALs, maxLogs=32 2023-07-12 15:39:43,122 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40285,DS-81fd12ba-0a3e-4a96-814c-a95c78be509b,DISK] 2023-07-12 15:39:43,127 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/WALs/jenkins-hbase5.apache.org,45089,1689176381369/jenkins-hbase5.apache.org%2C45089%2C1689176381369.meta.1689176383107.meta 2023-07-12 15:39:43,127 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40285,DS-81fd12ba-0a3e-4a96-814c-a95c78be509b,DISK]] 2023-07-12 15:39:43,128 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:39:43,130 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 15:39:43,150 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 15:39:43,154 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 15:39:43,160 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 15:39:43,160 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:43,160 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 15:39:43,160 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 15:39:43,163 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 15:39:43,165 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/info 2023-07-12 15:39:43,165 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/info 2023-07-12 15:39:43,165 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 15:39:43,166 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:43,167 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 15:39:43,168 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:39:43,168 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/rep_barrier 2023-07-12 15:39:43,169 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 15:39:43,169 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:43,170 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 15:39:43,171 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/table 2023-07-12 15:39:43,171 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/table 2023-07-12 15:39:43,172 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 15:39:43,172 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:43,174 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740 2023-07-12 15:39:43,177 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740 2023-07-12 15:39:43,180 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 15:39:43,182 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 15:39:43,184 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74894461, jitterRate=0.11601443588733673}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 15:39:43,184 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 15:39:43,193 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689176383076 2023-07-12 15:39:43,209 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 15:39:43,210 INFO [RS_OPEN_META-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 15:39:43,210 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase5.apache.org,45089,1689176381369, state=OPEN 2023-07-12 15:39:43,213 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 15:39:43,213 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 15:39:43,217 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 15:39:43,217 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase5.apache.org,45089,1689176381369 in 313 msec 2023-07-12 15:39:43,221 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 15:39:43,222 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 515 msec 2023-07-12 15:39:43,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 810 msec 2023-07-12 15:39:43,227 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689176383226, completionTime=-1 2023-07-12 15:39:43,227 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 15:39:43,227 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 15:39:43,291 DEBUG [hconnection-0x6d3d5c49-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:43,294 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:48180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:43,312 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 15:39:43,312 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689176443312 2023-07-12 15:39:43,312 INFO [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689176503312 2023-07-12 15:39:43,312 INFO [master/jenkins-hbase5:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-07-12 15:39:43,337 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37087,1689176380200-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:43,337 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37087,1689176380200-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:43,337 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37087,1689176380200-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:43,339 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase5:37087, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:43,340 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:43,345 DEBUG [master/jenkins-hbase5:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 15:39:43,354 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 15:39:43,356 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 15:39:43,366 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 15:39:43,368 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:39:43,371 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:39:43,392 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:43,394 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8 empty. 2023-07-12 15:39:43,395 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:43,395 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 15:39:43,815 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 15:39:43,817 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => a8c1301b634cdb9d0c9d3c8a94ed47b8, NAME => 'hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:39:44,230 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:44,230 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing a8c1301b634cdb9d0c9d3c8a94ed47b8, disabling compactions & flushes 2023-07-12 15:39:44,230 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:39:44,230 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:39:44,230 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. after waiting 0 ms 2023-07-12 15:39:44,230 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:39:44,230 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:39:44,230 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for a8c1301b634cdb9d0c9d3c8a94ed47b8: 2023-07-12 15:39:44,235 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:39:44,250 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689176384238"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176384238"}]},"ts":"1689176384238"} 2023-07-12 15:39:44,275 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:39:44,277 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:39:44,281 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176384277"}]},"ts":"1689176384277"} 2023-07-12 15:39:44,284 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 15:39:44,296 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a8c1301b634cdb9d0c9d3c8a94ed47b8, ASSIGN}] 2023-07-12 15:39:44,299 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a8c1301b634cdb9d0c9d3c8a94ed47b8, ASSIGN 2023-07-12 15:39:44,300 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a8c1301b634cdb9d0c9d3c8a94ed47b8, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:39:44,451 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a8c1301b634cdb9d0c9d3c8a94ed47b8, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:44,452 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689176384451"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176384451"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176384451"}]},"ts":"1689176384451"} 2023-07-12 15:39:44,456 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure a8c1301b634cdb9d0c9d3c8a94ed47b8, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:39:44,616 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:39:44,616 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a8c1301b634cdb9d0c9d3c8a94ed47b8, NAME => 'hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:39:44,617 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:44,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:44,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:44,618 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:44,620 INFO [StoreOpener-a8c1301b634cdb9d0c9d3c8a94ed47b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:44,622 DEBUG [StoreOpener-a8c1301b634cdb9d0c9d3c8a94ed47b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8/info 2023-07-12 15:39:44,622 DEBUG [StoreOpener-a8c1301b634cdb9d0c9d3c8a94ed47b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8/info 2023-07-12 15:39:44,622 INFO [StoreOpener-a8c1301b634cdb9d0c9d3c8a94ed47b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8c1301b634cdb9d0c9d3c8a94ed47b8 columnFamilyName info 2023-07-12 15:39:44,623 INFO [StoreOpener-a8c1301b634cdb9d0c9d3c8a94ed47b8-1] regionserver.HStore(310): Store=a8c1301b634cdb9d0c9d3c8a94ed47b8/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:44,625 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:44,626 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:44,630 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:39:44,633 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:39:44,634 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened a8c1301b634cdb9d0c9d3c8a94ed47b8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67705971, jitterRate=0.008897587656974792}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:39:44,634 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for a8c1301b634cdb9d0c9d3c8a94ed47b8: 2023-07-12 15:39:44,636 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8., pid=6, masterSystemTime=1689176384609 2023-07-12 15:39:44,640 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:39:44,640 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:39:44,641 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a8c1301b634cdb9d0c9d3c8a94ed47b8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:44,642 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689176384641"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176384641"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176384641"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176384641"}]},"ts":"1689176384641"} 2023-07-12 15:39:44,649 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 15:39:44,649 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure a8c1301b634cdb9d0c9d3c8a94ed47b8, server=jenkins-hbase5.apache.org,45089,1689176381369 in 190 msec 2023-07-12 15:39:44,652 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 15:39:44,652 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a8c1301b634cdb9d0c9d3c8a94ed47b8, ASSIGN in 353 msec 2023-07-12 15:39:44,654 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:39:44,654 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176384654"}]},"ts":"1689176384654"} 2023-07-12 15:39:44,657 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 15:39:44,660 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:39:44,662 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3030 sec 2023-07-12 15:39:44,669 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 15:39:44,671 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:39:44,671 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:39:44,704 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 15:39:44,721 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:39:44,727 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-07-12 15:39:44,738 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 15:39:44,751 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:39:44,755 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-07-12 15:39:44,765 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 15:39:44,768 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 15:39:44,770 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.268sec 2023-07-12 15:39:44,771 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(103): Quota table not found. Creating... 2023-07-12 15:39:44,772 INFO [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:quota', {NAME => 'q', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'u', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:39:44,773 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:quota 2023-07-12 15:39:44,773 INFO [master/jenkins-hbase5:0:becomeActiveMaster] quotas.MasterQuotaManager(107): Initializing quota support 2023-07-12 15:39:44,775 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:39:44,775 INFO [master/jenkins-hbase5:0:becomeActiveMaster] namespace.NamespaceStateManager(59): Namespace State Manager started. 2023-07-12 15:39:44,777 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:39:44,779 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:44,780 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b empty. 2023-07-12 15:39:44,781 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:44,781 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:quota regions 2023-07-12 15:39:44,782 INFO [master/jenkins-hbase5:0:becomeActiveMaster] namespace.NamespaceStateManager(222): Finished updating state of 2 namespaces. 2023-07-12 15:39:44,782 INFO [master/jenkins-hbase5:0:becomeActiveMaster] namespace.NamespaceAuditor(50): NamespaceAuditor started. 2023-07-12 15:39:44,784 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=QuotaObserverChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:44,785 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=QuotaObserverChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:44,786 INFO [master/jenkins-hbase5:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 15:39:44,786 INFO [master/jenkins-hbase5:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 15:39:44,787 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37087,1689176380200-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 15:39:44,788 INFO [master/jenkins-hbase5:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase5.apache.org,37087,1689176380200-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 15:39:44,802 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 15:39:44,874 DEBUG [Listener at localhost/38241] zookeeper.ReadOnlyZKClient(139): Connect 0x0971df84 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:44,879 DEBUG [Listener at localhost/38241] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22e7bd1f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:44,891 DEBUG [hconnection-0x7fa05a00-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:44,902 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:48182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:44,912 INFO [Listener at localhost/38241] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:39:44,913 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [10,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:39:44,929 INFO [regionserver/jenkins-hbase5:0.Chore.1] quotas.SpaceQuotaRefresherChore(80): Quota table not found, skipping quota manager cache refresh. 2023-07-12 15:39:45,200 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/hbase/quota/.tabledesc/.tableinfo.0000000001 2023-07-12 15:39:45,202 INFO [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(7675): creating {ENCODED => a5559313088d907a5c06d3ba77e10f9b, NAME => 'hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:quota', {NAME => 'q', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'u', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:39:45,615 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(866): Instantiated hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:45,615 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1604): Closing a5559313088d907a5c06d3ba77e10f9b, disabling compactions & flushes 2023-07-12 15:39:45,615 INFO [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1626): Closing region hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:39:45,615 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:39:45,615 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. after waiting 0 ms 2023-07-12 15:39:45,615 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:39:45,615 INFO [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1838): Closed hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:39:45,615 DEBUG [RegionOpenAndInit-hbase:quota-pool-0] regionserver.HRegion(1558): Region close journal for a5559313088d907a5c06d3ba77e10f9b: 2023-07-12 15:39:45,620 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:39:45,622 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b.","families":{"info":[{"qualifier":"regioninfo","vlen":37,"tag":[],"timestamp":"1689176385622"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176385622"}]},"ts":"1689176385622"} 2023-07-12 15:39:45,625 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:39:45,626 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:39:45,626 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:quota","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176385626"}]},"ts":"1689176385626"} 2023-07-12 15:39:45,628 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:quota, state=ENABLING in hbase:meta 2023-07-12 15:39:45,634 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:quota, region=a5559313088d907a5c06d3ba77e10f9b, ASSIGN}] 2023-07-12 15:39:45,636 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:quota, region=a5559313088d907a5c06d3ba77e10f9b, ASSIGN 2023-07-12 15:39:45,637 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:quota, region=a5559313088d907a5c06d3ba77e10f9b, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:39:45,655 INFO [Listener at localhost/38241] hbase.ResourceChecker(147): before: quotas.TestSpaceQuotasWithSnapshots#testRematerializedTablesDoNoInheritSpace Thread=211, OpenFileDescriptor=473, MaxFileDescriptor=60000, SystemLoadAverage=189, ProcessCount=166, AvailableMemoryMB=7344 2023-07-12 15:39:45,667 DEBUG [Listener at localhost/38241] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 15:39:45,670 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49988, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 15:39:45,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testRematerializedTablesDoNoInheritSpace0', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:39:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=11, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:45,685 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=11, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRematerializedTablesDoNoInheritSpace0 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:39:45,687 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=11, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRematerializedTablesDoNoInheritSpace0 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:39:45,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testRematerializedTablesDoNoInheritSpace0" procId is: 11 2023-07-12 15:39:45,689 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:45,690 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c empty. 2023-07-12 15:39:45,691 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:45,691 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRematerializedTablesDoNoInheritSpace0 regions 2023-07-12 15:39:45,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=11 2023-07-12 15:39:45,789 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a5559313088d907a5c06d3ba77e10f9b, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:45,789 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b.","families":{"info":[{"qualifier":"regioninfo","vlen":37,"tag":[],"timestamp":"1689176385788"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176385788"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176385788"}]},"ts":"1689176385788"} 2023-07-12 15:39:45,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=12, ppid=10, state=RUNNABLE; OpenRegionProcedure a5559313088d907a5c06d3ba77e10f9b, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:39:45,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=11 2023-07-12 15:39:45,909 DEBUG [regionserver/jenkins-hbase5:0.Chore.1] client.ConnectionImplementation(1060): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 46 failed; retrying after sleep of 46 org.apache.hadoop.hbase.client.NoServerForRegionException: No server address listed in hbase:meta for region hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. containing row t. at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:1029) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:866) at org.apache.hadoop.hbase.client.ServerConnectionUtils$ShortCircuitingClusterConnection.locateRegion(ServerConnectionUtils.java:70) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:157) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:266) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:309) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:594) at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:51) at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapshotsFromQuotaTable(SpaceQuotaRefresherChore.java:186) at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQuotaRefresherChore.java:91) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:39:45,949 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:39:45,949 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a5559313088d907a5c06d3ba77e10f9b, NAME => 'hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:39:45,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table quota a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:45,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:45,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:45,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:45,952 INFO [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family q of region a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:45,954 DEBUG [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/q 2023-07-12 15:39:45,954 DEBUG [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/q 2023-07-12 15:39:45,954 INFO [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5559313088d907a5c06d3ba77e10f9b columnFamilyName q 2023-07-12 15:39:45,955 INFO [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] regionserver.HStore(310): Store=a5559313088d907a5c06d3ba77e10f9b/q, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:45,956 INFO [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family u of region a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:45,958 DEBUG [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/u 2023-07-12 15:39:45,958 DEBUG [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/u 2023-07-12 15:39:45,958 INFO [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5559313088d907a5c06d3ba77e10f9b columnFamilyName u 2023-07-12 15:39:45,959 INFO [StoreOpener-a5559313088d907a5c06d3ba77e10f9b-1] regionserver.HStore(310): Store=a5559313088d907a5c06d3ba77e10f9b/u, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:45,961 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:45,962 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:45,965 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:quota descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 15:39:45,967 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:39:45,970 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:39:45,971 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened a5559313088d907a5c06d3ba77e10f9b; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65249398, jitterRate=-0.027708202600479126}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 15:39:45,971 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for a5559313088d907a5c06d3ba77e10f9b: 2023-07-12 15:39:45,973 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b., pid=12, masterSystemTime=1689176385945 2023-07-12 15:39:45,975 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:39:45,975 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:39:45,976 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a5559313088d907a5c06d3ba77e10f9b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:45,976 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b.","families":{"info":[{"qualifier":"regioninfo","vlen":37,"tag":[],"timestamp":"1689176385976"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176385976"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176385976"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176385976"}]},"ts":"1689176385976"} 2023-07-12 15:39:45,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=10 2023-07-12 15:39:45,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=10, state=SUCCESS; OpenRegionProcedure a5559313088d907a5c06d3ba77e10f9b, server=jenkins-hbase5.apache.org,45089,1689176381369 in 187 msec 2023-07-12 15:39:45,985 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 15:39:45,985 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=hbase:quota, region=a5559313088d907a5c06d3ba77e10f9b, ASSIGN in 348 msec 2023-07-12 15:39:45,986 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:39:45,987 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:quota","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176385986"}]},"ts":"1689176385986"} 2023-07-12 15:39:45,989 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:quota, state=ENABLED in hbase:meta 2023-07-12 15:39:45,992 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:quota execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:39:45,994 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=hbase:quota in 1.2200 sec 2023-07-12 15:39:46,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=11 2023-07-12 15:39:46,108 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testRematerializedTablesDoNoInheritSpace0/.tabledesc/.tableinfo.0000000001 2023-07-12 15:39:46,111 INFO [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(7675): creating {ENCODED => b5c1ee8ec2de1c08a223d44b92d4447c, NAME => 'testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRematerializedTablesDoNoInheritSpace0', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:39:46,123 DEBUG [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(866): Instantiated testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:46,123 DEBUG [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(1604): Closing b5c1ee8ec2de1c08a223d44b92d4447c, disabling compactions & flushes 2023-07-12 15:39:46,123 INFO [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(1626): Closing region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:46,123 DEBUG [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:46,123 DEBUG [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(1714): Acquired close lock on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. after waiting 0 ms 2023-07-12 15:39:46,123 DEBUG [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(1724): Updates disabled for region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:46,123 INFO [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(1838): Closed testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:46,123 DEBUG [RegionOpenAndInit-testRematerializedTablesDoNoInheritSpace0-pool-0] regionserver.HRegion(1558): Region close journal for b5c1ee8ec2de1c08a223d44b92d4447c: 2023-07-12 15:39:46,127 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=11, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRematerializedTablesDoNoInheritSpace0 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:39:46,128 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1689176386128"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176386128"}]},"ts":"1689176386128"} 2023-07-12 15:39:46,131 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:39:46,132 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=11, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRematerializedTablesDoNoInheritSpace0 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:39:46,132 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRematerializedTablesDoNoInheritSpace0","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176386132"}]},"ts":"1689176386132"} 2023-07-12 15:39:46,134 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRematerializedTablesDoNoInheritSpace0, state=ENABLING in hbase:meta 2023-07-12 15:39:46,137 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=11, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRematerializedTablesDoNoInheritSpace0, region=b5c1ee8ec2de1c08a223d44b92d4447c, ASSIGN}] 2023-07-12 15:39:46,139 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=11, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRematerializedTablesDoNoInheritSpace0, region=b5c1ee8ec2de1c08a223d44b92d4447c, ASSIGN 2023-07-12 15:39:46,140 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=11, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRematerializedTablesDoNoInheritSpace0, region=b5c1ee8ec2de1c08a223d44b92d4447c, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:39:46,291 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b5c1ee8ec2de1c08a223d44b92d4447c, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:46,291 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1689176386291"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176386291"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176386291"}]},"ts":"1689176386291"} 2023-07-12 15:39:46,294 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure b5c1ee8ec2de1c08a223d44b92d4447c, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:39:46,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=11 2023-07-12 15:39:46,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:46,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b5c1ee8ec2de1c08a223d44b92d4447c, NAME => 'testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:39:46,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRematerializedTablesDoNoInheritSpace0 b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:46,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:46,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:46,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:46,456 INFO [StoreOpener-b5c1ee8ec2de1c08a223d44b92d4447c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:46,458 DEBUG [StoreOpener-b5c1ee8ec2de1c08a223d44b92d4447c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/f1 2023-07-12 15:39:46,458 DEBUG [StoreOpener-b5c1ee8ec2de1c08a223d44b92d4447c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/f1 2023-07-12 15:39:46,459 INFO [StoreOpener-b5c1ee8ec2de1c08a223d44b92d4447c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b5c1ee8ec2de1c08a223d44b92d4447c columnFamilyName f1 2023-07-12 15:39:46,460 INFO [StoreOpener-b5c1ee8ec2de1c08a223d44b92d4447c-1] regionserver.HStore(310): Store=b5c1ee8ec2de1c08a223d44b92d4447c/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:46,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:46,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:46,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:46,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:39:46,471 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened b5c1ee8ec2de1c08a223d44b92d4447c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69474989, jitterRate=0.035258010029792786}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:39:46,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for b5c1ee8ec2de1c08a223d44b92d4447c: 2023-07-12 15:39:46,472 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c., pid=14, masterSystemTime=1689176386447 2023-07-12 15:39:46,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:46,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:46,476 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b5c1ee8ec2de1c08a223d44b92d4447c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:46,476 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1689176386475"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176386475"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176386475"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176386475"}]},"ts":"1689176386475"} 2023-07-12 15:39:46,481 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 15:39:46,482 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure b5c1ee8ec2de1c08a223d44b92d4447c, server=jenkins-hbase5.apache.org,45089,1689176381369 in 184 msec 2023-07-12 15:39:46,484 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=11 2023-07-12 15:39:46,485 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=11, state=SUCCESS; TransitRegionStateProcedure table=testRematerializedTablesDoNoInheritSpace0, region=b5c1ee8ec2de1c08a223d44b92d4447c, ASSIGN in 345 msec 2023-07-12 15:39:46,486 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=11, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRematerializedTablesDoNoInheritSpace0 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:39:46,486 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRematerializedTablesDoNoInheritSpace0","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176386486"}]},"ts":"1689176386486"} 2023-07-12 15:39:46,488 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRematerializedTablesDoNoInheritSpace0, state=ENABLED in hbase:meta 2023-07-12 15:39:46,491 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=11, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRematerializedTablesDoNoInheritSpace0 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:39:46,493 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, state=SUCCESS; CreateTableProcedure table=testRematerializedTablesDoNoInheritSpace0 in 809 msec 2023-07-12 15:39:46,804 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x66a026ba to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:46,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=11 2023-07-12 15:39:46,808 INFO [Listener at localhost/38241] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRematerializedTablesDoNoInheritSpace0, procId: 11 completed 2023-07-12 15:39:46,808 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(360): Writing data 2023-07-12 15:39:46,809 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6afab51f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:46,811 DEBUG [hconnection-0x496c7fb4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:46,813 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:46,921 DEBUG [regionserver/jenkins-hbase5:0.Chore.1] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 15:39:46,923 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:51960, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=MasterService 2023-07-12 15:39:47,401 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 15:39:47,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:39:47,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,423 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRematerializedTablesDoNoInheritSpace0' 2023-07-12 15:39:47,423 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:39:47,424 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRematerializedTablesDoNoInheritSpace0' starting 'acquire' 2023-07-12 15:39:47,424 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRematerializedTablesDoNoInheritSpace0', kicking off acquire phase on members. 2023-07-12 15:39:47,425 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,425 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,426 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:39:47,426 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:39:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:39:47,427 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,427 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:39:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:39:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,430 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRematerializedTablesDoNoInheritSpace0' with timeout 60000ms 2023-07-12 15:39:47,430 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:39:47,431 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' starting 'acquire' stage 2023-07-12 15:39:47,431 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:39:47,432 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:39:47,432 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:47,432 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. started... 2023-07-12 15:39:47,433 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5c1ee8ec2de1c08a223d44b92d4447c 1/1 column families, dataSize=2.22 MB heapSize=2.81 MB 2023-07-12 15:39:47,566 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.22 MB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/.tmp/f1/be32d077c1a44a2391a7fbfa749d17e3 2023-07-12 15:39:47,613 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/.tmp/f1/be32d077c1a44a2391a7fbfa749d17e3 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/f1/be32d077c1a44a2391a7fbfa749d17e3 2023-07-12 15:39:47,624 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/f1/be32d077c1a44a2391a7fbfa749d17e3, entries=8192, sequenceid=165, filesize=2.3 M 2023-07-12 15:39:47,626 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.22 MB/2325418, heapSize ~2.81 MB/2949360, currentSize=0 B/0 for b5c1ee8ec2de1c08a223d44b92d4447c in 193ms, sequenceid=165, compaction requested=false 2023-07-12 15:39:47,628 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRematerializedTablesDoNoInheritSpace0' 2023-07-12 15:39:47,631 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5c1ee8ec2de1c08a223d44b92d4447c: 2023-07-12 15:39:47,631 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:47,631 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:39:47,632 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:39:47,632 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,632 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' locally acquired 2023-07-12 15:39:47,632 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testRematerializedTablesDoNoInheritSpace0) in zk 2023-07-12 15:39:47,635 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,635 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:47,636 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,636 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:39:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:47,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testRematerializedTablesDoNoInheritSpace0' on coordinator 2023-07-12 15:39:47,638 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRematerializedTablesDoNoInheritSpace0' starting 'in-barrier' execution. 2023-07-12 15:39:47,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555f9b28[Count = 0] remaining members to acquire global barrier 2023-07-12 15:39:47,638 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,645 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,645 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' received 'reached' from coordinator. 2023-07-12 15:39:47,645 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' locally completed 2023-07-12 15:39:47,645 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRematerializedTablesDoNoInheritSpace0' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:39:47,645 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,645 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:39:47,648 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' has notified controller of completion 2023-07-12 15:39:47,648 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,648 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:39:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:47,648 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' completed. 2023-07-12 15:39:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRematerializedTablesDoNoInheritSpace0' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:39:47,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testRematerializedTablesDoNoInheritSpace0', counting down latch. Waiting for 0 more 2023-07-12 15:39:47,653 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRematerializedTablesDoNoInheritSpace0' execution completed 2023-07-12 15:39:47,653 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:39:47,653 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:39:47,653 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,653 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRematerializedTablesDoNoInheritSpace0including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:39:47,655 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,655 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:47,656 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:39:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:39:47,657 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:39:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,659 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,667 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,668 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,668 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,668 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:39:47,668 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:47,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:39:47,668 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:39:47,668 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:39:47,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRematerializedTablesDoNoInheritSpace0' 2023-07-12 15:39:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:39:47,668 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:39:47,669 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,669 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:47,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:39:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:39:47,670 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRematerializedTablesDoNoInheritSpace0'' to complete. (max 20000 ms per retry) 2023-07-12 15:39:47,671 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:39:47,771 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:39:47,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:39:47,778 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 15:39:47,778 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(372): Waiting until table size reflects written data 2023-07-12 15:39:47,779 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:39:47,789 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: null 2023-07-12 15:39:47,804 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testRematerializedTablesDoNoInheritSpace0 moved into observance of table space quota. 2023-07-12 15:39:47,805 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3cb24a53 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:47,839 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66a026ba to 127.0.0.1:51623 2023-07-12 15:39:47,839 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:39:47,844 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5158a2d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:47,846 DEBUG [hconnection-0x49973506-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:47,848 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37808, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:48,294 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:39:48,670 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 15:39:48,779 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 15:39:48,780 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 15:39:48,781 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:quota' 2023-07-12 15:39:48,796 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:39:48,859 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testRematerializedTablesDoNoInheritSpace0 moved into observance of table space quota. 2023-07-12 15:39:48,860 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3879b067 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:48,864 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69b37f52, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:48,866 DEBUG [hconnection-0x4a50864d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:48,868 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37824, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:49,298 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:39:49,299 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [15,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:39:49,300 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:39:49,300 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=-1 2023-07-12 15:39:49,799 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x20e50169 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:49,803 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:39:49,803 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:39:49,804 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fd5ff9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:49,806 DEBUG [hconnection-0x419311cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:49,807 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37838, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:50,305 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:39:50,305 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:39:50,366 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.quotas.MasterQuotasObserver 2023-07-12 15:39:50,366 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.quotas.MasterQuotasObserver Metrics about HBase MasterObservers 2023-07-12 15:39:50,367 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:39:50,367 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 15:39:50,807 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:39:50,807 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:39:50,812 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3153379f to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:50,816 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f44318b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:50,818 DEBUG [hconnection-0x1bf088d7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:50,820 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37842, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:51,309 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:39:51,309 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:39:51,810 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:39:51,811 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:39:51,812 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(386): Initial table size was 2359562 2023-07-12 15:39:51,812 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(388): Snapshot the table 2023-07-12 15:39:51,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } 2023-07-12 15:39:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689176391822 (current time:1689176391822). 2023-07-12 15:39:51,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 15:39:51,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] zookeeper.ReadOnlyZKClient(139): Connect 0x70feba1a to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:51,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@689ab18b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:51,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:51,834 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37850, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:51,837 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x31a704e3 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70feba1a to 127.0.0.1:51623 2023-07-12 15:39:51,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:39:51,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 15:39:51,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } 2023-07-12 15:39:51,864 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5eaa2b47, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:51,868 DEBUG [hconnection-0x2c5cb18e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:51,869 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37858, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=EXCLUSIVE 2023-07-12 15:39:51,884 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=15, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=EXCLUSIVE 2023-07-12 15:39:51,885 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=15, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=EXCLUSIVE; timeout=600000, timestamp=1689176991885 2023-07-12 15:39:51,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(653): Started snapshot: { ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } 2023-07-12 15:39:51,885 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot testRematerializedTablesDoNoInheritSpace0_snapshot1 C_M_SNAPSHOT_TABLE on table testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:51,886 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=15, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=EXCLUSIVE 2023-07-12 15:39:51,887 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=SHARED 2023-07-12 15:39:51,888 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=EXCLUSIVE in 10 msec 2023-07-12 15:39:51,888 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=16, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=SHARED 2023-07-12 15:39:51,889 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=16, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=SHARED; timeout=600000, timestamp=1689176991889 2023-07-12 15:39:51,890 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 15:39:51,890 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 15:39:51,990 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:39:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } is done 2023-07-12 15:39:51,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:39:51,994 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 15:39:52,195 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:39:52,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } is done 2023-07-12 15:39:52,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:39:52,196 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 15:39:52,304 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,304 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' 2023-07-12 15:39:52,304 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 15:39:52,304 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' starting 'acquire' 2023-07-12 15:39:52,304 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1', kicking off acquire phase on members. 2023-07-12 15:39:52,305 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,305 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,311 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 15:39:52,311 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 15:39:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:39:52,311 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,311 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:39:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 113 2023-07-12 15:39:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,312 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testRematerializedTablesDoNoInheritSpace0_snapshot1 from table testRematerializedTablesDoNoInheritSpace0 type FLUSH 2023-07-12 15:39:52,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,314 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' with timeout 300000ms 2023-07-12 15:39:52,314 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 15:39:52,314 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' starting 'acquire' stage 2023-07-12 15:39:52,314 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' locally acquired 2023-07-12 15:39:52,314 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testRematerializedTablesDoNoInheritSpace0_snapshot1) in zk 2023-07-12 15:39:52,317 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,317 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:39:52,318 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,318 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:39:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:52,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' on coordinator 2023-07-12 15:39:52,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a0011a[Count = 0] remaining members to acquire global barrier 2023-07-12 15:39:52,320 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' starting 'in-barrier' execution. 2023-07-12 15:39:52,320 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,321 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,322 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' received 'reached' from coordinator. 2023-07-12 15:39:52,322 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,322 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:39:52,322 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 15:39:52,322 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 15:39:52,322 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:52,323 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. started... 2023-07-12 15:39:52,323 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for b5c1ee8ec2de1c08a223d44b92d4447c: 2023-07-12 15:39:52,324 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c.' region-info for snapshot=testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,330 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:39:52,334 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/f1/be32d077c1a44a2391a7fbfa749d17e3] hfiles 2023-07-12 15:39:52,334 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/f1/be32d077c1a44a2391a7fbfa749d17e3 for snapshot=testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,496 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:39:52,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } is done 2023-07-12 15:39:52,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:39:52,498 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 15:39:52,750 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. completed. 2023-07-12 15:39:52,750 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:52,751 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 15:39:52,751 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 15:39:52,751 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,751 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' locally completed 2023-07-12 15:39:52,751 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:39:52,753 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,753 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' has notified controller of completion 2023-07-12 15:39:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:39:52,754 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:39:52,754 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' completed. 2023-07-12 15:39:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:39:52,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testRematerializedTablesDoNoInheritSpace0_snapshot1', counting down latch. Waiting for 0 more 2023-07-12 15:39:52,758 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRematerializedTablesDoNoInheritSpace0_snapshot1' execution completed 2023-07-12 15:39:52,758 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:39:52,758 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:39:52,758 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,758 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRematerializedTablesDoNoInheritSpace0_snapshot1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 15:39:52,761 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,761 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:39:52,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,761 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 15:39:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:52,762 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 15:39:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:39:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,764 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,768 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,768 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,768 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 15:39:52,768 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 15:39:52,768 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:39:52,768 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:52,768 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:39:52,768 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 15:39:52,768 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,768 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,769 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,769 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 15:39:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:39:52,771 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:52,823 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x210e6c9d to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:52,829 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79ff40d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:52,830 DEBUG [hconnection-0x7430e04b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:52,832 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37870, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:52,998 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:39:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } is done 2023-07-12 15:39:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:39:53,000 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 15:39:53,205 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testRematerializedTablesDoNoInheritSpace0_snapshot1 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:53,792 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4f2642f8 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:53,796 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19c0cf71, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:53,798 DEBUG [hconnection-0xe41a95d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:53,800 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:54,000 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:39:54,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } is done 2023-07-12 15:39:54,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:39:54,002 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#6) Sleeping: 2000ms while waiting for snapshot completion. 2023-07-12 15:39:54,046 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testRematerializedTablesDoNoInheritSpace0_snapshot1 of table testRematerializedTablesDoNoInheritSpace0 completed 2023-07-12 15:39:54,046 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:54,048 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:54,048 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot testRematerializedTablesDoNoInheritSpace0_snapshot1 C_M_SNAPSHOT_TABLE on table testRematerializedTablesDoNoInheritSpace0 at 1689176391885Consolidate snapshot: testRematerializedTablesDoNoInheritSpace0_snapshot1 at 1689176392768 (+883 ms)Loading Region manifests for testRematerializedTablesDoNoInheritSpace0_snapshot1 at 1689176392769 (+1 ms)Writing data manifest for testRematerializedTablesDoNoInheritSpace0_snapshot1 at 1689176392781 (+12 ms)Verifying snapshot: testRematerializedTablesDoNoInheritSpace0_snapshot1 at 1689176393194 (+413 ms)Snapshot testRematerializedTablesDoNoInheritSpace0_snapshot1 of table testRematerializedTablesDoNoInheritSpace0 completed at 1689176394046 (+852 ms) 2023-07-12 15:39:54,049 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=16, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=SHARED 2023-07-12 15:39:54,050 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRematerializedTablesDoNoInheritSpace0, type=SHARED in 2.1630 sec 2023-07-12 15:39:54,871 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x22a0cb3c to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:54,875 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64073001, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:54,877 DEBUG [hconnection-0x332d2c39-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:54,878 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37890, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:55,822 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x690f2901 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:55,827 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c196681, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:55,829 DEBUG [hconnection-0x240ba54a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:55,830 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:56,002 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:39:56,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 } is done 2023-07-12 15:39:56,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(401): Snapshot '{ ss=testRematerializedTablesDoNoInheritSpace0_snapshot1 table=testRematerializedTablesDoNoInheritSpace0 type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 15:39:56,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRematerializedTablesDoNoInheritSpace1 snapshot=name: "testRematerializedTablesDoNoInheritSpace0_snapshot1" table: "testRematerializedTablesDoNoInheritSpace0" creation_time: 1689176391822 type: FLUSH version: 2 ttl: 0 ) 2023-07-12 15:39:56,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(750): Clone snapshot=testRematerializedTablesDoNoInheritSpace0_snapshot1 as table=testRematerializedTablesDoNoInheritSpace1 2023-07-12 15:39:56,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=17 2023-07-12 15:39:56,050 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testRematerializedTablesDoNoInheritSpace1/.tabledesc/.tableinfo.0000000001 2023-07-12 15:39:56,056 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "testRematerializedTablesDoNoInheritSpace0_snapshot1" table: "testRematerializedTablesDoNoInheritSpace0" creation_time: 1689176391822 type: FLUSH version: 2 ttl: 0 2023-07-12 15:39:56,056 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testRematerializedTablesDoNoInheritSpace1 2023-07-12 15:39:56,057 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(239): region to add: b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:39:56,057 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(585): clone region=b5c1ee8ec2de1c08a223d44b92d4447c as 98647cced3a0461ae7cfd8e4946afc6b in snapshot testRematerializedTablesDoNoInheritSpace0_snapshot1 2023-07-12 15:39:56,058 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7675): creating {ENCODED => 98647cced3a0461ae7cfd8e4946afc6b, NAME => 'testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRematerializedTablesDoNoInheritSpace1', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:39:56,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=17 2023-07-12 15:39:56,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=17 2023-07-12 15:39:56,469 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(866): Instantiated testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:56,471 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink be32d077c1a44a2391a7fbfa749d17e3 from cloned region in snapshot testRematerializedTablesDoNoInheritSpace0_snapshot1 to table=testRematerializedTablesDoNoInheritSpace1 2023-07-12 15:39:56,477 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1604): Closing 98647cced3a0461ae7cfd8e4946afc6b, disabling compactions & flushes 2023-07-12 15:39:56,477 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1626): Closing region testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:39:56,477 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:39:56,477 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1714): Acquired close lock on testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. after waiting 0 ms 2023-07-12 15:39:56,477 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1724): Updates disabled for region testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:39:56,477 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1838): Closed testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:39:56,477 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1558): Region close journal for 98647cced3a0461ae7cfd8e4946afc6b: 2023-07-12 15:39:56,478 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "testRematerializedTablesDoNoInheritSpace0_snapshot1" table: "testRematerializedTablesDoNoInheritSpace0" creation_time: 1689176391822 type: FLUSH version: 2 ttl: 0 2023-07-12 15:39:56,478 INFO [PEWorker-5] procedure.CloneSnapshotProcedure$1(421): Clone snapshot=testRematerializedTablesDoNoInheritSpace0_snapshot1 on table=testRematerializedTablesDoNoInheritSpace1 completed! 2023-07-12 15:39:56,483 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1689176396483"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176396483"}]},"ts":"1689176396483"} 2023-07-12 15:39:56,485 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:39:56,486 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRematerializedTablesDoNoInheritSpace1","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176396486"}]},"ts":"1689176396486"} 2023-07-12 15:39:56,488 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRematerializedTablesDoNoInheritSpace1, state=ENABLING in hbase:meta 2023-07-12 15:39:56,492 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRematerializedTablesDoNoInheritSpace1, region=98647cced3a0461ae7cfd8e4946afc6b, ASSIGN}] 2023-07-12 15:39:56,494 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=18, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRematerializedTablesDoNoInheritSpace1, region=98647cced3a0461ae7cfd8e4946afc6b, ASSIGN 2023-07-12 15:39:56,495 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=18, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRematerializedTablesDoNoInheritSpace1, region=98647cced3a0461ae7cfd8e4946afc6b, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:39:56,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=17 2023-07-12 15:39:56,646 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=18 updating hbase:meta row=98647cced3a0461ae7cfd8e4946afc6b, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:56,646 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1689176396646"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176396646"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176396646"}]},"ts":"1689176396646"} 2023-07-12 15:39:56,648 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; OpenRegionProcedure 98647cced3a0461ae7cfd8e4946afc6b, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:39:56,789 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4d755e7c to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:56,794 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3095885a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:56,796 DEBUG [hconnection-0x551f4f6e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:56,798 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:56,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:39:56,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 98647cced3a0461ae7cfd8e4946afc6b, NAME => 'testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:39:56,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRematerializedTablesDoNoInheritSpace1 98647cced3a0461ae7cfd8e4946afc6b 2023-07-12 15:39:56,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:39:56,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 98647cced3a0461ae7cfd8e4946afc6b 2023-07-12 15:39:56,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 98647cced3a0461ae7cfd8e4946afc6b 2023-07-12 15:39:56,811 INFO [StoreOpener-98647cced3a0461ae7cfd8e4946afc6b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 98647cced3a0461ae7cfd8e4946afc6b 2023-07-12 15:39:56,813 DEBUG [StoreOpener-98647cced3a0461ae7cfd8e4946afc6b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/f1 2023-07-12 15:39:56,813 DEBUG [StoreOpener-98647cced3a0461ae7cfd8e4946afc6b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/f1 2023-07-12 15:39:56,814 INFO [StoreOpener-98647cced3a0461ae7cfd8e4946afc6b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98647cced3a0461ae7cfd8e4946afc6b columnFamilyName f1 2023-07-12 15:39:56,828 DEBUG [StoreOpener-98647cced3a0461ae7cfd8e4946afc6b-1] regionserver.HStore(539): loaded hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/f1/testRematerializedTablesDoNoInheritSpace0=b5c1ee8ec2de1c08a223d44b92d4447c-be32d077c1a44a2391a7fbfa749d17e3 2023-07-12 15:39:56,828 INFO [StoreOpener-98647cced3a0461ae7cfd8e4946afc6b-1] regionserver.HStore(310): Store=98647cced3a0461ae7cfd8e4946afc6b/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:39:56,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b 2023-07-12 15:39:56,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b 2023-07-12 15:39:56,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 98647cced3a0461ae7cfd8e4946afc6b 2023-07-12 15:39:56,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/recovered.edits/165.seqid, newMaxSeqId=165, maxSeqId=-1 2023-07-12 15:39:56,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 98647cced3a0461ae7cfd8e4946afc6b; next sequenceid=166; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63570740, jitterRate=-0.052722156047821045}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:39:56,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 98647cced3a0461ae7cfd8e4946afc6b: 2023-07-12 15:39:56,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b., pid=19, masterSystemTime=1689176396801 2023-07-12 15:39:56,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:39:56,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:39:56,842 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=18 updating hbase:meta row=98647cced3a0461ae7cfd8e4946afc6b, regionState=OPEN, openSeqNum=166, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:56,843 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1689176396842"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176396842"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176396842"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176396842"}]},"ts":"1689176396842"} 2023-07-12 15:39:56,846 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-12 15:39:56,847 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; OpenRegionProcedure 98647cced3a0461ae7cfd8e4946afc6b, server=jenkins-hbase5.apache.org,45089,1689176381369 in 196 msec 2023-07-12 15:39:56,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-12 15:39:56,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=testRematerializedTablesDoNoInheritSpace1, region=98647cced3a0461ae7cfd8e4946afc6b, ASSIGN in 355 msec 2023-07-12 15:39:56,850 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRematerializedTablesDoNoInheritSpace1","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176396850"}]},"ts":"1689176396850"} 2023-07-12 15:39:56,852 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRematerializedTablesDoNoInheritSpace1, state=ENABLED in hbase:meta 2023-07-12 15:39:56,856 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; CloneSnapshotProcedure (table=testRematerializedTablesDoNoInheritSpace1 snapshot=name: "testRematerializedTablesDoNoInheritSpace0_snapshot1" table: "testRematerializedTablesDoNoInheritSpace0" creation_time: 1689176391822 type: FLUSH version: 2 ttl: 0 ) in 833 msec 2023-07-12 15:39:57,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=17 2023-07-12 15:39:57,140 INFO [Listener at localhost/38241] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRematerializedTablesDoNoInheritSpace1, procId: 17 completed 2023-07-12 15:39:57,487 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 15:39:57,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:39:57,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure testRematerializedTablesDoNoInheritSpace0 was in running list but was completed. Accepting new attempt. 2023-07-12 15:39:57,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,494 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRematerializedTablesDoNoInheritSpace0' 2023-07-12 15:39:57,494 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:39:57,495 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRematerializedTablesDoNoInheritSpace0' starting 'acquire' 2023-07-12 15:39:57,495 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRematerializedTablesDoNoInheritSpace0', kicking off acquire phase on members. 2023-07-12 15:39:57,495 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,495 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,498 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:39:57,498 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:39:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:39:57,498 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,498 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:39:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:39:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,500 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRematerializedTablesDoNoInheritSpace0' with timeout 60000ms 2023-07-12 15:39:57,500 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:39:57,500 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' starting 'acquire' stage 2023-07-12 15:39:57,501 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:39:57,501 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:39:57,501 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:57,501 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. started... 2023-07-12 15:39:57,501 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b5c1ee8ec2de1c08a223d44b92d4447c 1/1 column families, dataSize=2.22 MB heapSize=2.81 MB 2023-07-12 15:39:57,546 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.22 MB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/.tmp/f1/49cab31dddb2484d988d8568de6ae96f 2023-07-12 15:39:57,556 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/.tmp/f1/49cab31dddb2484d988d8568de6ae96f as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/f1/49cab31dddb2484d988d8568de6ae96f 2023-07-12 15:39:57,565 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/f1/49cab31dddb2484d988d8568de6ae96f, entries=8192, sequenceid=330, filesize=2.3 M 2023-07-12 15:39:57,567 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.22 MB/2325418, heapSize ~2.81 MB/2949360, currentSize=0 B/0 for b5c1ee8ec2de1c08a223d44b92d4447c in 66ms, sequenceid=330, compaction requested=false 2023-07-12 15:39:57,567 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5c1ee8ec2de1c08a223d44b92d4447c: 2023-07-12 15:39:57,568 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:57,568 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:39:57,568 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:39:57,568 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,568 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' locally acquired 2023-07-12 15:39:57,569 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testRematerializedTablesDoNoInheritSpace0) in zk 2023-07-12 15:39:57,572 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,572 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:57,573 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,573 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:39:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:57,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testRematerializedTablesDoNoInheritSpace0' on coordinator 2023-07-12 15:39:57,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55603c5b[Count = 0] remaining members to acquire global barrier 2023-07-12 15:39:57,575 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRematerializedTablesDoNoInheritSpace0' starting 'in-barrier' execution. 2023-07-12 15:39:57,575 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,577 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,577 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' received 'reached' from coordinator. 2023-07-12 15:39:57,577 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' locally completed 2023-07-12 15:39:57,577 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,577 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:39:57,577 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRematerializedTablesDoNoInheritSpace0' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:39:57,579 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' has notified controller of completion 2023-07-12 15:39:57,579 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,580 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:39:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:57,580 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' completed. 2023-07-12 15:39:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRematerializedTablesDoNoInheritSpace0' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:39:57,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testRematerializedTablesDoNoInheritSpace0', counting down latch. Waiting for 0 more 2023-07-12 15:39:57,584 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRematerializedTablesDoNoInheritSpace0' execution completed 2023-07-12 15:39:57,584 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:39:57,584 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:39:57,584 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,584 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRematerializedTablesDoNoInheritSpace0including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:39:57,586 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,586 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:57,586 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:39:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:57,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:39:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:39:57,587 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,589 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,594 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,594 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,594 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,594 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,594 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,594 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,594 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,594 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:39:57,595 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:39:57,595 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:39:57,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRematerializedTablesDoNoInheritSpace0' 2023-07-12 15:39:57,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:39:57,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:39:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:39:57,596 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRematerializedTablesDoNoInheritSpace0'' to complete. (max 20000 ms per retry) 2023-07-12 15:39:57,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:39:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:39:57,596 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:39:57,696 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:39:57,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:39:57,698 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 15:39:57,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:39:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure testRematerializedTablesDoNoInheritSpace0 was in running list but was completed. Accepting new attempt. 2023-07-12 15:39:57,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,704 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRematerializedTablesDoNoInheritSpace0' 2023-07-12 15:39:57,705 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:39:57,705 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRematerializedTablesDoNoInheritSpace0' starting 'acquire' 2023-07-12 15:39:57,705 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRematerializedTablesDoNoInheritSpace0', kicking off acquire phase on members. 2023-07-12 15:39:57,706 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,706 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,707 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:39:57,707 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:39:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:39:57,708 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,708 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:39:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:39:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRematerializedTablesDoNoInheritSpace0 is still present, removing 2023-07-12 15:39:57,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,709 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRematerializedTablesDoNoInheritSpace0' with timeout 60000ms 2023-07-12 15:39:57,709 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:39:57,710 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' starting 'acquire' stage 2023-07-12 15:39:57,710 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:39:57,710 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:39:57,710 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:57,710 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. started... 2023-07-12 15:39:57,710 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b5c1ee8ec2de1c08a223d44b92d4447c: 2023-07-12 15:39:57,710 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:39:57,710 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:39:57,710 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:39:57,710 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,710 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' locally acquired 2023-07-12 15:39:57,710 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testRematerializedTablesDoNoInheritSpace0) in zk 2023-07-12 15:39:57,715 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,715 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:57,715 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,715 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:39:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:57,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testRematerializedTablesDoNoInheritSpace0' on coordinator 2023-07-12 15:39:57,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785c8642[Count = 0] remaining members to acquire global barrier 2023-07-12 15:39:57,717 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRematerializedTablesDoNoInheritSpace0' starting 'in-barrier' execution. 2023-07-12 15:39:57,717 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,720 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,720 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' received 'reached' from coordinator. 2023-07-12 15:39:57,720 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' locally completed 2023-07-12 15:39:57,720 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,720 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:39:57,720 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRematerializedTablesDoNoInheritSpace0' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:39:57,723 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,723 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' has notified controller of completion 2023-07-12 15:39:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:57,723 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:39:57,724 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRematerializedTablesDoNoInheritSpace0' completed. 2023-07-12 15:39:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRematerializedTablesDoNoInheritSpace0' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:39:57,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testRematerializedTablesDoNoInheritSpace0', counting down latch. Waiting for 0 more 2023-07-12 15:39:57,727 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRematerializedTablesDoNoInheritSpace0' execution completed 2023-07-12 15:39:57,727 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:39:57,727 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:39:57,727 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,727 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRematerializedTablesDoNoInheritSpace0including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:39:57,729 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,729 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,729 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:39:57,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:39:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:39:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:39:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:39:57,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:39:57,730 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:39:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,732 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:39:57,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,736 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,736 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:39:57,736 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:39:57,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:39:57,736 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:39:57,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 because node does not exist (not an error) 2023-07-12 15:39:57,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRematerializedTablesDoNoInheritSpace0' 2023-07-12 15:39:57,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:39:57,736 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,736 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:39:57,736 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:39:57,737 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRematerializedTablesDoNoInheritSpace0'' to complete. (max 20000 ms per retry) 2023-07-12 15:39:57,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:39:57,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:39:57,737 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,737 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:39:57,737 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,738 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRematerializedTablesDoNoInheritSpace0 2023-07-12 15:39:57,800 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testRematerializedTablesDoNoInheritSpace1 moved into observance of table space quota. 2023-07-12 15:39:57,801 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x13246c9f to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:57,812 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c52addb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:57,815 DEBUG [hconnection-0x4f7f774b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:57,817 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:57,838 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:39:57,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:39:57,840 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:39:57,842 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:39:58,786 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3c25527f to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:58,790 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@476ffb92, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:58,792 DEBUG [hconnection-0x4bf4fe98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:58,793 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:39:58,832 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testRematerializedTablesDoNoInheritSpace0 moved into observance of table space quota. 2023-07-12 15:39:58,843 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace0: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=4.51 MB/1 GB] 2023-07-12 15:39:58,847 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-07-12 15:39:58,849 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): 98647cced3a0461ae7cfd8e4946afc6b/f1 is initiating major compaction (all files) 2023-07-12 15:39:58,849 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:39:58,849 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:39:58,849 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of 98647cced3a0461ae7cfd8e4946afc6b/f1 in testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:39:58,849 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/f1/testRematerializedTablesDoNoInheritSpace0=b5c1ee8ec2de1c08a223d44b92d4447c-be32d077c1a44a2391a7fbfa749d17e3] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/.tmp, totalSize=2.3 M 2023-07-12 15:39:58,851 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting testRematerializedTablesDoNoInheritSpace0=b5c1ee8ec2de1c08a223d44b92d4447c-be32d077c1a44a2391a7fbfa749d17e3, keycount=8192, bloomtype=ROW, size=2.3 M, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1689176386891 2023-07-12 15:39:58,941 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): 98647cced3a0461ae7cfd8e4946afc6b#f1#compaction#2 average throughput is 29.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:39:59,358 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/.tmp/f1/030b30db4f0a4a2484ef2de48f027d54 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/f1/030b30db4f0a4a2484ef2de48f027d54 2023-07-12 15:39:59,372 DEBUG [Listener at localhost/38241] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRematerializedTablesDoNoInheritSpace1' 2023-07-12 15:39:59,373 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 98647cced3a0461ae7cfd8e4946afc6b/f1 of 98647cced3a0461ae7cfd8e4946afc6b into 030b30db4f0a4a2484ef2de48f027d54(size=2.3 M), total size for store is 2.3 M. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:39:59,373 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for 98647cced3a0461ae7cfd8e4946afc6b: 2023-07-12 15:39:59,373 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:39:59,374 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace1: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:39:59,786 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x10a4ac4d to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:39:59,792 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41e14942, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:39:59,794 DEBUG [hconnection-0x2fcf95ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:39:59,798 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:00,376 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace1: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:40:00,799 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1f3b7228 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:00,803 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a21ea39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:00,805 DEBUG [hconnection-0x8acdd5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:00,806 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:00,847 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testRematerializedTablesDoNoInheritSpace1 moved into observance of table space quota. 2023-07-12 15:40:01,378 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testRematerializedTablesDoNoInheritSpace1: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:01,397 INFO [Listener at localhost/38241] hbase.ResourceChecker(175): after: quotas.TestSpaceQuotasWithSnapshots#testRematerializedTablesDoNoInheritSpace Thread=275 (was 211) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x690f2901-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3153379f sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6d3d5c49-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6d3d5c49-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4f2642f8-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4d755e7c sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6d3d5c49-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4f2642f8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_741846745_17 at /127.0.0.1:59556 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x709b2855-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x13246c9f sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3c25527f-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x6d3d5c49-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x1f3b7228-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x31a704e3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2fcf95ae-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4d755e7c-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: (jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x20e50169 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x210e6c9d-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6d3d5c49-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1644750562_17 at /127.0.0.1:59422 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x1f3b7228-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x709b2855-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x20e50169-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x690f2901-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x20e50169-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3c25527f-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8acdd5-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/cluster_25c97bd4-4232-2ad9-6328-0c863aed8e02/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6d3d5c49-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x10a4ac4d-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x4bf4fe98-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/cluster_25c97bd4-4232-2ad9-6328-0c863aed8e02/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3153379f-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x31a704e3-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3879b067 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8acdd5-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x210e6c9d-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x2fcf95ae-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x13246c9f-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3cb24a53 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3879b067-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3cb24a53-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x709b2855-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3153379f-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x210e6c9d sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_741846745_17 at /127.0.0.1:59416 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x13246c9f-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4f2642f8-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x22a0cb3c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x1f3b7228 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4d755e7c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x10a4ac4d-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x22a0cb3c-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x4bf4fe98-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x690f2901 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x31a704e3-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x10a4ac4d sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3c25527f sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x22a0cb3c sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3cb24a53-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3879b067-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) - Thread LEAK? -, OpenFileDescriptor=587 (was 473) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=147 (was 189), ProcessCount=166 (was 166), AvailableMemoryMB=7100 (was 7344) 2023-07-12 15:40:01,410 INFO [Listener at localhost/38241] hbase.ResourceChecker(147): before: quotas.TestSpaceQuotasWithSnapshots#testNamespacesInheritSnapshotSize Thread=275, OpenFileDescriptor=587, MaxFileDescriptor=60000, SystemLoadAverage=147, ProcessCount=166, AvailableMemoryMB=7100 2023-07-12 15:40:01,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.HMaster$15(3014): Client=jenkins//172.31.10.131 creating {NAME => 'ns2'} 2023-07-12 15:40:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=20, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=ns2 2023-07-12 15:40:01,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=20 2023-07-12 15:40:01,425 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 15:40:01,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, state=SUCCESS; CreateNamespaceProcedure, namespace=ns2 in 12 msec 2023-07-12 15:40:01,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=20 2023-07-12 15:40:01,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'ns2:testNamespacesInheritSnapshotSize3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:40:01,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=21, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:01,530 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=21, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=ns2:testNamespacesInheritSnapshotSize3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:40:01,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "ns2" qualifier: "testNamespacesInheritSnapshotSize3" procId is: 21 2023-07-12 15:40:01,531 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=21, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=ns2:testNamespacesInheritSnapshotSize3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:40:01,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 15:40:01,533 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:01,534 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012 empty. 2023-07-12 15:40:01,535 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:01,535 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived ns2:testNamespacesInheritSnapshotSize3 regions 2023-07-12 15:40:01,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 15:40:01,816 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x76d911e8 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:01,823 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5651ed2e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:01,826 DEBUG [hconnection-0x8e5ab47-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:01,831 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:01,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 15:40:01,950 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/ns2/testNamespacesInheritSnapshotSize3/.tabledesc/.tableinfo.0000000001 2023-07-12 15:40:01,952 INFO [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3d092a61087068ae644d6d865e43b012, NAME => 'ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='ns2:testNamespacesInheritSnapshotSize3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:02,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 15:40:02,351 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 15:40:02,364 DEBUG [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(866): Instantiated ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:02,364 DEBUG [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(1604): Closing 3d092a61087068ae644d6d865e43b012, disabling compactions & flushes 2023-07-12 15:40:02,364 INFO [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(1626): Closing region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:02,364 DEBUG [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:02,364 DEBUG [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(1714): Acquired close lock on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. after waiting 0 ms 2023-07-12 15:40:02,364 DEBUG [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(1724): Updates disabled for region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:02,364 INFO [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(1838): Closed ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:02,364 DEBUG [RegionOpenAndInit-ns2:testNamespacesInheritSnapshotSize3-pool-0] regionserver.HRegion(1558): Region close journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:02,367 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=21, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=ns2:testNamespacesInheritSnapshotSize3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:40:02,369 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689176402368"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176402368"}]},"ts":"1689176402368"} 2023-07-12 15:40:02,370 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:40:02,371 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=21, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=ns2:testNamespacesInheritSnapshotSize3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:40:02,371 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"ns2:testNamespacesInheritSnapshotSize3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176402371"}]},"ts":"1689176402371"} 2023-07-12 15:40:02,373 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=ns2:testNamespacesInheritSnapshotSize3, state=ENABLING in hbase:meta 2023-07-12 15:40:02,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=ns2:testNamespacesInheritSnapshotSize3, region=3d092a61087068ae644d6d865e43b012, ASSIGN}] 2023-07-12 15:40:02,378 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=ns2:testNamespacesInheritSnapshotSize3, region=3d092a61087068ae644d6d865e43b012, ASSIGN 2023-07-12 15:40:02,379 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=22, ppid=21, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=ns2:testNamespacesInheritSnapshotSize3, region=3d092a61087068ae644d6d865e43b012, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:02,530 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=3d092a61087068ae644d6d865e43b012, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:02,530 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689176402529"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176402529"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176402529"}]},"ts":"1689176402529"} 2023-07-12 15:40:02,532 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; OpenRegionProcedure 3d092a61087068ae644d6d865e43b012, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:02,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 15:40:02,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:02,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3d092a61087068ae644d6d865e43b012, NAME => 'ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:40:02,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testNamespacesInheritSnapshotSize3 3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:02,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:02,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:02,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:02,690 INFO [StoreOpener-3d092a61087068ae644d6d865e43b012-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:02,692 DEBUG [StoreOpener-3d092a61087068ae644d6d865e43b012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1 2023-07-12 15:40:02,692 DEBUG [StoreOpener-3d092a61087068ae644d6d865e43b012-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1 2023-07-12 15:40:02,693 INFO [StoreOpener-3d092a61087068ae644d6d865e43b012-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d092a61087068ae644d6d865e43b012 columnFamilyName f1 2023-07-12 15:40:02,693 INFO [StoreOpener-3d092a61087068ae644d6d865e43b012-1] regionserver.HStore(310): Store=3d092a61087068ae644d6d865e43b012/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:02,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:02,695 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:02,698 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:02,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:02,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 3d092a61087068ae644d6d865e43b012; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66876705, jitterRate=-0.003459438681602478}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:02,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:02,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012., pid=23, masterSystemTime=1689176402684 2023-07-12 15:40:02,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:02,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:02,704 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=3d092a61087068ae644d6d865e43b012, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:02,704 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689176402704"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176402704"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176402704"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176402704"}]},"ts":"1689176402704"} 2023-07-12 15:40:02,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 15:40:02,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; OpenRegionProcedure 3d092a61087068ae644d6d865e43b012, server=jenkins-hbase5.apache.org,45089,1689176381369 in 174 msec 2023-07-12 15:40:02,709 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-07-12 15:40:02,710 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; TransitRegionStateProcedure table=ns2:testNamespacesInheritSnapshotSize3, region=3d092a61087068ae644d6d865e43b012, ASSIGN in 332 msec 2023-07-12 15:40:02,711 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=21, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=ns2:testNamespacesInheritSnapshotSize3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:40:02,711 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"ns2:testNamespacesInheritSnapshotSize3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176402711"}]},"ts":"1689176402711"} 2023-07-12 15:40:02,712 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=ns2:testNamespacesInheritSnapshotSize3, state=ENABLED in hbase:meta 2023-07-12 15:40:02,715 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=21, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=ns2:testNamespacesInheritSnapshotSize3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:40:02,717 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, state=SUCCESS; CreateTableProcedure table=ns2:testNamespacesInheritSnapshotSize3 in 1.1880 sec 2023-07-12 15:40:02,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x14d37f35 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:02,790 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e13532a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:02,792 DEBUG [hconnection-0x321795a9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:02,795 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:02,864 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/f1/testRematerializedTablesDoNoInheritSpace0=b5c1ee8ec2de1c08a223d44b92d4447c-be32d077c1a44a2391a7fbfa749d17e3] to archive 2023-07-12 15:40:02,865 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:02,872 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/f1/testRematerializedTablesDoNoInheritSpace0=b5c1ee8ec2de1c08a223d44b92d4447c-be32d077c1a44a2391a7fbfa749d17e3 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/f1/testRematerializedTablesDoNoInheritSpace0=b5c1ee8ec2de1c08a223d44b92d4447c-be32d077c1a44a2391a7fbfa749d17e3 2023-07-12 15:40:03,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=21 2023-07-12 15:40:03,640 INFO [Listener at localhost/38241] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: ns2:testNamespacesInheritSnapshotSize3, procId: 21 completed 2023-07-12 15:40:03,641 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(205): Writing data 2023-07-12 15:40:03,787 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4daa16f6 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:03,792 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21486df8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:03,795 DEBUG [hconnection-0x5fb14841-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:03,796 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:03,851 INFO [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(402): ns2:testNamespacesInheritSnapshotSize3 moving into observance of namespace space quota 2023-07-12 15:40:03,920 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 15:40:03,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:03,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,926 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:03,926 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:03,927 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'acquire' 2023-07-12 15:40:03,927 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'ns2:testNamespacesInheritSnapshotSize3', kicking off acquire phase on members. 2023-07-12 15:40:03,927 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,927 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,929 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:03,929 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:03,929 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,929 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:03,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'ns2:testNamespacesInheritSnapshotSize3' with timeout 60000ms 2023-07-12 15:40:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'acquire' stage 2023-07-12 15:40:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:40:03,931 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:03,932 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:03,932 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. started... 2023-07-12 15:40:03,932 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d092a61087068ae644d6d865e43b012 1/1 column families, dataSize=2.22 MB heapSize=2.81 MB 2023-07-12 15:40:03,967 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.22 MB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/.tmp/f1/715924a40a0143c7a1809c80918ef41d 2023-07-12 15:40:03,975 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/.tmp/f1/715924a40a0143c7a1809c80918ef41d as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/715924a40a0143c7a1809c80918ef41d 2023-07-12 15:40:03,982 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/715924a40a0143c7a1809c80918ef41d, entries=8192, sequenceid=165, filesize=2.3 M 2023-07-12 15:40:03,984 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.22 MB/2325418, heapSize ~2.81 MB/2949360, currentSize=0 B/0 for 3d092a61087068ae644d6d865e43b012 in 52ms, sequenceid=165, compaction requested=false 2023-07-12 15:40:03,984 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:03,985 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:03,985 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:03,985 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:40:03,986 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:40:03,986 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,986 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' locally acquired 2023-07-12 15:40:03,986 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (ns2:testNamespacesInheritSnapshotSize3) in zk 2023-07-12 15:40:03,989 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,990 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:03,990 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,990 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:03,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'ns2:testNamespacesInheritSnapshotSize3' on coordinator 2023-07-12 15:40:03,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5173d4ba[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:03,992 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'in-barrier' execution. 2023-07-12 15:40:03,992 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,993 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,993 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' received 'reached' from coordinator. 2023-07-12 15:40:03,993 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' locally completed 2023-07-12 15:40:03,993 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'ns2:testNamespacesInheritSnapshotSize3' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:03,993 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,993 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:03,995 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' has notified controller of completion 2023-07-12 15:40:03,995 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,995 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:03,995 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' completed. 2023-07-12 15:40:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'ns2:testNamespacesInheritSnapshotSize3' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:03,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'ns2:testNamespacesInheritSnapshotSize3', counting down latch. Waiting for 0 more 2023-07-12 15:40:03,998 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'ns2:testNamespacesInheritSnapshotSize3' execution completed 2023-07-12 15:40:03,999 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:03,999 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:03,999 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:03,999 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure ns2:testNamespacesInheritSnapshotSize3including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:04,000 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,000 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,000 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:04,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:04,001 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,003 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,006 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,007 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,007 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:04,007 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:04,007 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:04,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:04,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:04,007 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,007 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:04,008 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,008 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:04,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:04,008 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : ns2:testNamespacesInheritSnapshotSize3'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:04,008 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:04,008 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,108 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:04,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:04,109 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 15:40:04,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:04,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure ns2:testNamespacesInheritSnapshotSize3 was in running list but was completed. Accepting new attempt. 2023-07-12 15:40:04,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,115 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:04,115 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:04,115 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'acquire' 2023-07-12 15:40:04,115 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'ns2:testNamespacesInheritSnapshotSize3', kicking off acquire phase on members. 2023-07-12 15:40:04,116 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,116 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,117 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:04,117 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:04,117 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,117 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc ns2:testNamespacesInheritSnapshotSize3 is still present, removing 2023-07-12 15:40:04,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,119 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'ns2:testNamespacesInheritSnapshotSize3' with timeout 60000ms 2023-07-12 15:40:04,119 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:04,119 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'acquire' stage 2023-07-12 15:40:04,119 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:40:04,119 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:04,119 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:04,119 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. started... 2023-07-12 15:40:04,120 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:04,120 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:04,120 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:40:04,120 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:40:04,120 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,120 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' locally acquired 2023-07-12 15:40:04,120 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (ns2:testNamespacesInheritSnapshotSize3) in zk 2023-07-12 15:40:04,121 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,121 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:04,122 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,122 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:04,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'ns2:testNamespacesInheritSnapshotSize3' on coordinator 2023-07-12 15:40:04,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51dbb75f[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:04,123 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'in-barrier' execution. 2023-07-12 15:40:04,123 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,128 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,128 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' received 'reached' from coordinator. 2023-07-12 15:40:04,129 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' locally completed 2023-07-12 15:40:04,129 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'ns2:testNamespacesInheritSnapshotSize3' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:04,129 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,129 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:04,130 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,130 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' has notified controller of completion 2023-07-12 15:40:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:04,130 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:04,130 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' completed. 2023-07-12 15:40:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'ns2:testNamespacesInheritSnapshotSize3' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:04,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'ns2:testNamespacesInheritSnapshotSize3', counting down latch. Waiting for 0 more 2023-07-12 15:40:04,133 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'ns2:testNamespacesInheritSnapshotSize3' execution completed 2023-07-12 15:40:04,133 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:04,133 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:04,133 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,133 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure ns2:testNamespacesInheritSnapshotSize3including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:04,135 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,135 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:04,135 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:04,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:04,135 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,137 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,139 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,139 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,139 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:04,139 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:04,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:04,139 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:04,139 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:04,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:04,139 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:04,140 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : ns2:testNamespacesInheritSnapshotSize3'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:04,140 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:04,139 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 because node does not exist (not an error) 2023-07-12 15:40:04,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:04,140 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,140 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:04,240 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:04,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:04,242 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(216): Waiting until namespace size reflects written data 2023-07-12 15:40:04,242 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:04,246 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:40:04,748 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:40:04,836 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x6dee9e4f to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:04,841 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@248b9475, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:04,842 DEBUG [hconnection-0x2f42dea3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:04,844 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55152, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:05,249 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:40:05,751 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:40:05,809 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x327e884f to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:05,813 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620c41a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:05,815 DEBUG [hconnection-0x11558487-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:05,817 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:55168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:05,858 INFO [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(402): ns2:testNamespacesInheritSnapshotSize3 moving into observance of namespace space quota 2023-07-12 15:40:06,252 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:06,252 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [15,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:06,253 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:06,253 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=-1 2023-07-12 15:40:06,754 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:06,755 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:06,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4d5e04d2 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:06,790 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@157b561d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:06,791 DEBUG [hconnection-0x6aa8e2ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:06,792 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:07,256 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:07,256 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:07,757 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:07,758 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:07,877 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x644a9ab7 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:07,881 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ff678c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:07,883 DEBUG [hconnection-0x3b328a92-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:07,884 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:08,191 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 15:40:08,259 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:08,259 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:08,760 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:08,760 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:08,761 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(230): Initial table size was 2359562 2023-07-12 15:40:08,761 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(232): Snapshot the table 2023-07-12 15:40:08,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 } 2023-07-12 15:40:08,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689176408762 (current time:1689176408762). 2023-07-12 15:40:08,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 15:40:08,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] zookeeper.ReadOnlyZKClient(139): Connect 0x129be4bc to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:08,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51cbd7d5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:08,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:08,769 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:08,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x129be4bc to 127.0.0.1:51623 2023-07-12 15:40:08,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:08,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 15:40:08,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 } 2023-07-12 15:40:08,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=EXCLUSIVE 2023-07-12 15:40:08,774 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=24, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=EXCLUSIVE 2023-07-12 15:40:08,774 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=24, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=EXCLUSIVE; timeout=600000, timestamp=1689177008774 2023-07-12 15:40:08,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(653): Started snapshot: { ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 } 2023-07-12 15:40:08,774 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot testNamespacesInheritSnapshotSize3_snapshot1 C_M_SNAPSHOT_TABLE on table ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:08,775 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:08,775 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 15:40:08,775 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=24, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=EXCLUSIVE 2023-07-12 15:40:08,775 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=25, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=SHARED 2023-07-12 15:40:08,776 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=EXCLUSIVE in 3 msec 2023-07-12 15:40:08,776 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=25, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=SHARED 2023-07-12 15:40:08,777 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=25, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=SHARED; timeout=600000, timestamp=1689177008777 2023-07-12 15:40:08,807 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7a14df5d to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:08,812 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26b64c8a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:08,813 DEBUG [hconnection-0x4e9f6c94-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:08,815 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34084, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:08,875 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 } is done 2023-07-12 15:40:08,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:08,876 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 15:40:09,076 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:09,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 } is done 2023-07-12 15:40:09,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:09,078 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 15:40:09,187 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,187 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testNamespacesInheritSnapshotSize3_snapshot1' 2023-07-12 15:40:09,187 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 15:40:09,187 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testNamespacesInheritSnapshotSize3_snapshot1' starting 'acquire' 2023-07-12 15:40:09,187 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testNamespacesInheritSnapshotSize3_snapshot1', kicking off acquire phase on members. 2023-07-12 15:40:09,188 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,188 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,189 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 15:40:09,189 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 15:40:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:40:09,189 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,189 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 103 2023-07-12 15:40:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,190 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testNamespacesInheritSnapshotSize3_snapshot1 from table ns2:testNamespacesInheritSnapshotSize3 type FLUSH 2023-07-12 15:40:09,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,191 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testNamespacesInheritSnapshotSize3_snapshot1' with timeout 300000ms 2023-07-12 15:40:09,191 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 15:40:09,191 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testNamespacesInheritSnapshotSize3_snapshot1' starting 'acquire' stage 2023-07-12 15:40:09,191 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testNamespacesInheritSnapshotSize3_snapshot1' locally acquired 2023-07-12 15:40:09,191 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testNamespacesInheritSnapshotSize3_snapshot1) in zk 2023-07-12 15:40:09,192 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,192 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:09,193 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,193 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testNamespacesInheritSnapshotSize3_snapshot1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:09,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:09,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testNamespacesInheritSnapshotSize3_snapshot1' on coordinator 2023-07-12 15:40:09,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58dcbdb[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:09,194 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testNamespacesInheritSnapshotSize3_snapshot1' starting 'in-barrier' execution. 2023-07-12 15:40:09,194 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,196 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,196 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testNamespacesInheritSnapshotSize3_snapshot1' received 'reached' from coordinator. 2023-07-12 15:40:09,196 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,196 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:09,196 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 15:40:09,196 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 15:40:09,196 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:09,196 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. started... 2023-07-12 15:40:09,196 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:09,196 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012.' region-info for snapshot=testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,197 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:09,197 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/715924a40a0143c7a1809c80918ef41d] hfiles 2023-07-12 15:40:09,197 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/715924a40a0143c7a1809c80918ef41d for snapshot=testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,378 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:09,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 } is done 2023-07-12 15:40:09,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:09,379 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 15:40:09,606 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. completed. 2023-07-12 15:40:09,606 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:09,606 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 15:40:09,606 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 15:40:09,606 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,606 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testNamespacesInheritSnapshotSize3_snapshot1' locally completed 2023-07-12 15:40:09,606 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testNamespacesInheritSnapshotSize3_snapshot1' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:09,609 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,609 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testNamespacesInheritSnapshotSize3_snapshot1' has notified controller of completion 2023-07-12 15:40:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:09,609 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:09,609 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testNamespacesInheritSnapshotSize3_snapshot1' completed. 2023-07-12 15:40:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:09,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testNamespacesInheritSnapshotSize3_snapshot1' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:09,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testNamespacesInheritSnapshotSize3_snapshot1', counting down latch. Waiting for 0 more 2023-07-12 15:40:09,612 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testNamespacesInheritSnapshotSize3_snapshot1' execution completed 2023-07-12 15:40:09,612 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:09,612 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:09,612 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,612 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testNamespacesInheritSnapshotSize3_snapshot1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 15:40:09,613 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,613 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:09,614 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 15:40:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:09,614 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 15:40:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:40:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,615 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,617 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,617 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 15:40:09,617 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 15:40:09,618 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:40:09,618 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,618 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:09,618 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,618 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 15:40:09,619 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:09,619 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,619 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,619 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 15:40:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:40:09,619 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:09,800 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x02cf6116 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:09,806 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64e4bbe4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:09,807 DEBUG [hconnection-0x4fd6ee4a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:09,809 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:09,879 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:09,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 } is done 2023-07-12 15:40:09,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:09,881 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 15:40:10,039 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testNamespacesInheritSnapshotSize3_snapshot1 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:10,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3a63e856 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:10,790 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40fcbe91, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:10,792 DEBUG [hconnection-0x1669960c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:10,795 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:10,866 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testNamespacesInheritSnapshotSize3_snapshot1 of table ns2:testNamespacesInheritSnapshotSize3 completed 2023-07-12 15:40:10,866 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:10,867 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testNamespacesInheritSnapshotSize3_snapshot1 2023-07-12 15:40:10,867 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot testNamespacesInheritSnapshotSize3_snapshot1 C_M_SNAPSHOT_TABLE on table ns2:testNamespacesInheritSnapshotSize3 at 1689176408774Consolidate snapshot: testNamespacesInheritSnapshotSize3_snapshot1 at 1689176409618 (+844 ms)Loading Region manifests for testNamespacesInheritSnapshotSize3_snapshot1 at 1689176409618Writing data manifest for testNamespacesInheritSnapshotSize3_snapshot1 at 1689176409622 (+4 ms)Verifying snapshot: testNamespacesInheritSnapshotSize3_snapshot1 at 1689176410032 (+410 ms)Snapshot testNamespacesInheritSnapshotSize3_snapshot1 of table ns2:testNamespacesInheritSnapshotSize3 completed at 1689176410866 (+834 ms) 2023-07-12 15:40:10,869 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=25, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=SHARED 2023-07-12 15:40:10,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=ns2:testNamespacesInheritSnapshotSize3, type=SHARED in 2.0940 sec 2023-07-12 15:40:10,881 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:10,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 } is done 2023-07-12 15:40:10,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(401): Snapshot '{ ss=testNamespacesInheritSnapshotSize3_snapshot1 table=ns2:testNamespacesInheritSnapshotSize3 type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 15:40:10,882 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(238): Write more data 2023-07-12 15:40:11,090 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 15:40:11,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:11,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure ns2:testNamespacesInheritSnapshotSize3 was in running list but was completed. Accepting new attempt. 2023-07-12 15:40:11,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,096 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:11,096 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:11,096 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'acquire' 2023-07-12 15:40:11,096 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'ns2:testNamespacesInheritSnapshotSize3', kicking off acquire phase on members. 2023-07-12 15:40:11,097 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,097 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,098 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,098 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:11,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:11,099 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,099 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc ns2:testNamespacesInheritSnapshotSize3 is still present, removing 2023-07-12 15:40:11,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,100 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'ns2:testNamespacesInheritSnapshotSize3' with timeout 60000ms 2023-07-12 15:40:11,100 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:11,101 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'acquire' stage 2023-07-12 15:40:11,101 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:40:11,101 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:11,101 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:11,101 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. started... 2023-07-12 15:40:11,101 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d092a61087068ae644d6d865e43b012 1/1 column families, dataSize=2.22 MB heapSize=2.81 MB 2023-07-12 15:40:11,530 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.22 MB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/.tmp/f1/d6061791335d46fb8d08461486fa0395 2023-07-12 15:40:11,538 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/.tmp/f1/d6061791335d46fb8d08461486fa0395 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/d6061791335d46fb8d08461486fa0395 2023-07-12 15:40:11,545 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/d6061791335d46fb8d08461486fa0395, entries=8192, sequenceid=331, filesize=2.3 M 2023-07-12 15:40:11,546 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.22 MB/2325418, heapSize ~2.81 MB/2949360, currentSize=0 B/0 for 3d092a61087068ae644d6d865e43b012 in 445ms, sequenceid=331, compaction requested=false 2023-07-12 15:40:11,546 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:11,546 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:11,546 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:40:11,547 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:40:11,547 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,547 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' locally acquired 2023-07-12 15:40:11,547 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (ns2:testNamespacesInheritSnapshotSize3) in zk 2023-07-12 15:40:11,550 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,550 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:11,550 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,550 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:11,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'ns2:testNamespacesInheritSnapshotSize3' on coordinator 2023-07-12 15:40:11,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@197f33be[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:11,552 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'in-barrier' execution. 2023-07-12 15:40:11,552 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,553 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,553 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' received 'reached' from coordinator. 2023-07-12 15:40:11,553 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' locally completed 2023-07-12 15:40:11,553 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'ns2:testNamespacesInheritSnapshotSize3' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:11,553 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,553 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:11,555 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' has notified controller of completion 2023-07-12 15:40:11,555 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,555 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:11,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:11,555 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' completed. 2023-07-12 15:40:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'ns2:testNamespacesInheritSnapshotSize3' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:11,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'ns2:testNamespacesInheritSnapshotSize3', counting down latch. Waiting for 0 more 2023-07-12 15:40:11,558 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'ns2:testNamespacesInheritSnapshotSize3' execution completed 2023-07-12 15:40:11,558 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:11,558 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:11,559 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,559 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure ns2:testNamespacesInheritSnapshotSize3including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:11,560 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,560 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:11,561 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:11,561 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,562 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,569 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,569 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:11,569 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:11,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:11,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 because node does not exist (not an error) 2023-07-12 15:40:11,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:11,569 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:11,569 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,569 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:11,570 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,570 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,570 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,570 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : ns2:testNamespacesInheritSnapshotSize3'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:11,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:11,570 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,570 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:11,670 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:11,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:11,671 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 15:40:11,671 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(240): Flush the table 2023-07-12 15:40:11,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:11,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure ns2:testNamespacesInheritSnapshotSize3 was in running list but was completed. Accepting new attempt. 2023-07-12 15:40:11,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,677 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:11,677 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:11,677 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'acquire' 2023-07-12 15:40:11,677 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'ns2:testNamespacesInheritSnapshotSize3', kicking off acquire phase on members. 2023-07-12 15:40:11,678 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,678 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,679 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:11,679 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:11,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:11,680 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,680 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc ns2:testNamespacesInheritSnapshotSize3 is still present, removing 2023-07-12 15:40:11,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,681 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'ns2:testNamespacesInheritSnapshotSize3' with timeout 60000ms 2023-07-12 15:40:11,681 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:11,682 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'acquire' stage 2023-07-12 15:40:11,682 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:40:11,682 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:11,682 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:11,682 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. started... 2023-07-12 15:40:11,682 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:11,682 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:11,682 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:40:11,682 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:40:11,682 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,682 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' locally acquired 2023-07-12 15:40:11,682 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (ns2:testNamespacesInheritSnapshotSize3) in zk 2023-07-12 15:40:11,684 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,684 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:11,685 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,685 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:11,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'ns2:testNamespacesInheritSnapshotSize3' on coordinator 2023-07-12 15:40:11,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e6f22f[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:11,686 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'ns2:testNamespacesInheritSnapshotSize3' starting 'in-barrier' execution. 2023-07-12 15:40:11,686 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,688 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,688 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' received 'reached' from coordinator. 2023-07-12 15:40:11,688 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' locally completed 2023-07-12 15:40:11,688 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'ns2:testNamespacesInheritSnapshotSize3' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:11,688 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,688 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:11,690 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' has notified controller of completion 2023-07-12 15:40:11,690 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,690 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:11,690 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'ns2:testNamespacesInheritSnapshotSize3' completed. 2023-07-12 15:40:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'ns2:testNamespacesInheritSnapshotSize3' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:11,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'ns2:testNamespacesInheritSnapshotSize3', counting down latch. Waiting for 0 more 2023-07-12 15:40:11,695 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'ns2:testNamespacesInheritSnapshotSize3' execution completed 2023-07-12 15:40:11,695 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:11,695 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:11,695 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,695 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure ns2:testNamespacesInheritSnapshotSize3including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:11,696 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,696 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:11,697 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:11,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:11,697 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:11,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,700 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:11,703 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:11,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:11,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 because node does not exist (not an error) 2023-07-12 15:40:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:11,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'ns2:testNamespacesInheritSnapshotSize3' 2023-07-12 15:40:11,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/ns2:testNamespacesInheritSnapshotSize3 2023-07-12 15:40:11,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:11,703 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : ns2:testNamespacesInheritSnapshotSize3'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:11,704 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:11,797 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x41c0446c to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:11,803 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ca4cc13, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:11,804 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:11,804 DEBUG [hconnection-0x57c4b8ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:11,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:11,805 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(242): Synchronously compacting the table 2023-07-12 15:40:11,805 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:11,805 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): 3d092a61087068ae644d6d865e43b012/f1 is initiating major compaction (all files) 2023-07-12 15:40:11,805 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:11,805 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:11,805 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:11,806 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of 3d092a61087068ae644d6d865e43b012/f1 in ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:11,806 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/715924a40a0143c7a1809c80918ef41d, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/d6061791335d46fb8d08461486fa0395] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/.tmp, totalSize=4.5 M 2023-07-12 15:40:11,807 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 715924a40a0143c7a1809c80918ef41d, keycount=8192, bloomtype=ROW, size=2.3 M, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1689176403649 2023-07-12 15:40:11,808 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting d6061791335d46fb8d08461486fa0395, keycount=8192, bloomtype=ROW, size=2.3 M, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1689176410883 2023-07-12 15:40:11,904 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): 3d092a61087068ae644d6d865e43b012#f1#compaction#5 average throughput is 26.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:12,315 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/.tmp/f1/e3bf83c3f4a940d39376a928c74c72d2 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/e3bf83c3f4a940d39376a928c74c72d2 2023-07-12 15:40:12,321 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 3d092a61087068ae644d6d865e43b012/f1 of 3d092a61087068ae644d6d865e43b012 into e3bf83c3f4a940d39376a928c74c72d2(size=2.3 M), total size for store is 2.3 M. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:12,321 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:12,321 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(248): Waiting for the region reports to reflect the correct size, between (1585152, 2609152) 2023-07-12 15:40:12,322 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:12,331 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$7(254): Master observed table sizes from region size reports: {hbase:meta=0, hbase:namespace=0, testRematerializedTablesDoNoInheritSpace0=4729212, testRematerializedTablesDoNoInheritSpace1=2359672, ns2:testNamespacesInheritSnapshotSize3=4729263, hbase:quota=0} 2023-07-12 15:40:12,786 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x18da73fb to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:12,791 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4eb78e35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:12,792 DEBUG [hconnection-0x472a0f49-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:12,794 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34130, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:12,832 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$7(254): Master observed table sizes from region size reports: {hbase:meta=0, hbase:namespace=0, testRematerializedTablesDoNoInheritSpace0=4729212, testRematerializedTablesDoNoInheritSpace1=2359672, ns2:testNamespacesInheritSnapshotSize3=4729263, hbase:quota=0} 2023-07-12 15:40:12,846 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/715924a40a0143c7a1809c80918ef41d, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/d6061791335d46fb8d08461486fa0395] to archive 2023-07-12 15:40:12,847 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:12,849 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/715924a40a0143c7a1809c80918ef41d to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/715924a40a0143c7a1809c80918ef41d 2023-07-12 15:40:12,851 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/d6061791335d46fb8d08461486fa0395 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/f1/d6061791335d46fb8d08461486fa0395 2023-07-12 15:40:12,853 INFO [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(402): ns2:testNamespacesInheritSnapshotSize3 moving into observance of namespace space quota 2023-07-12 15:40:13,333 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$7(254): Master observed table sizes from region size reports: {hbase:meta=0, hbase:namespace=0, testRematerializedTablesDoNoInheritSpace0=4729212, testRematerializedTablesDoNoInheritSpace1=2359672, ns2:testNamespacesInheritSnapshotSize3=2369769, hbase:quota=0} 2023-07-12 15:40:13,335 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [15,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:13,335 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=-1 2023-07-12 15:40:13,795 INFO [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(402): ns2:testNamespacesInheritSnapshotSize3 moving into observance of namespace space quota 2023-07-12 15:40:13,797 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x0cd75564 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:13,801 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ff4eff1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:13,803 DEBUG [hconnection-0x63390a3d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:13,805 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:13,836 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369769 2023-07-12 15:40:14,337 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369769 2023-07-12 15:40:14,829 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5cbb07d0 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:14,833 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e78a033, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:14,834 DEBUG [hconnection-0x6c474d98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:14,835 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:14,839 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369769 2023-07-12 15:40:15,340 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369769 2023-07-12 15:40:15,811 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4daf8ad5 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:15,820 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c2a5aec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:15,822 DEBUG [hconnection-0x533e3757-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:15,823 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:34148, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:15,841 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369769 2023-07-12 15:40:15,843 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(267): Final observed size of table: 2369769 2023-07-12 15:40:15,844 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [20,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:15,844 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=4.51 MB/1 GB] 2023-07-12 15:40:15,845 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(282): Expecting namespace usage to be 2359562 + 2369769 = 4729331 2023-07-12 15:40:15,845 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:15,846 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for ns2: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=4.51 MB/1 GB] 2023-07-12 15:40:15,846 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$9(288): Checking for 4729331 == 4729331 2023-07-12 15:40:15,869 INFO [Listener at localhost/38241] hbase.ResourceChecker(175): after: quotas.TestSpaceQuotasWithSnapshots#testNamespacesInheritSnapshotSize Thread=318 (was 275) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3a63e856 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x7a14df5d-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x14d37f35-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4daa16f6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_741846745_17 at /127.0.0.1:42876 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4d5e04d2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x76d911e8-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x327e884f-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4d5e04d2-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x7fa05a00-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x533e3757-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x5cbb07d0-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x41c0446c sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x709b2855-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x63390a3d-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x0cd75564-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4daf8ad5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x76d911e8-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x18da73fb sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x63390a3d-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x41c0446c-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x6dee9e4f-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x533e3757-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3a63e856-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x327e884f sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x6dee9e4f-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x18da73fb-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x644a9ab7-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4daf8ad5-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x02cf6116 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x6dee9e4f sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x3a63e856-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x6c474d98-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4daa16f6-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x7fa05a00-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x18da73fb-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x41c0446c-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x644a9ab7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1644750562_17 at /127.0.0.1:59466 [Waiting for operation #35] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6c474d98-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x02cf6116-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x327e884f-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4daf8ad5-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x76d911e8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x644a9ab7-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x5cbb07d0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x7a14df5d-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4daa16f6-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x0cd75564 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x0cd75564-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: hconnection-0x709b2855-shared-pool-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x5cbb07d0-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x14d37f35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x02cf6116-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x709b2855-shared-pool-30 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1644750562_17 at /127.0.0.1:42908 [Waiting for operation #16] sun.nio.ch.FileChannelImpl.transferTo0(Native Method) sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:605) org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223) org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:280) org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:620) org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:804) org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:751) org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:609) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:145) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:100) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x7a14df5d sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/1537108297.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x4d5e04d2-SendThread(127.0.0.1:51623) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51623@0x14d37f35-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) - Thread LEAK? -, OpenFileDescriptor=699 (was 587) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=113 (was 147), ProcessCount=166 (was 166), AvailableMemoryMB=7023 (was 7100) 2023-07-12 15:40:15,885 INFO [Listener at localhost/38241] hbase.ResourceChecker(147): before: quotas.TestSpaceQuotasWithSnapshots#testTablesWithSnapshots Thread=318, OpenFileDescriptor=699, MaxFileDescriptor=60000, SystemLoadAverage=113, ProcessCount=166, AvailableMemoryMB=7018 2023-07-12 15:40:15,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:40:15,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=26, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testTablesWithSnapshots4 2023-07-12 15:40:15,892 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=26, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testTablesWithSnapshots4 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:40:15,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testTablesWithSnapshots4" procId is: 26 2023-07-12 15:40:15,893 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=26, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testTablesWithSnapshots4 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:40:15,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=26 2023-07-12 15:40:15,900 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:15,900 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70 2023-07-12 15:40:15,900 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:15,900 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70 empty. 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824 empty. 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-11] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d empty. 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81 empty. 2023-07-12 15:40:15,901 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd empty. 2023-07-12 15:40:15,902 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1 empty. 2023-07-12 15:40:15,902 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12 empty. 2023-07-12 15:40:15,902 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962 empty. 2023-07-12 15:40:15,902 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:15,902 DEBUG [HFileArchiver-9] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:15,902 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:15,902 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:15,903 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:15,903 DEBUG [HFileArchiver-11] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:15,903 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:15,903 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70 2023-07-12 15:40:15,903 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:15,903 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:15,903 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a empty. 2023-07-12 15:40:15,903 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e empty. 2023-07-12 15:40:15,904 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:15,904 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:15,904 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testTablesWithSnapshots4 regions 2023-07-12 15:40:15,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=26 2023-07-12 15:40:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=26 2023-07-12 15:40:16,320 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesWithSnapshots4/.tabledesc/.tableinfo.0000000001 2023-07-12 15:40:16,321 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 16362efeb81ea80068441d436c3937dd, NAME => 'testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd.', STARTKEY => '', ENDKEY => '0'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,322 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 964fa8dd605f4f75805eb52796edffe1, NAME => 'testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,322 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 7e64a90200e73af2749582231be39d70, NAME => 'testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70.', STARTKEY => '0', ENDKEY => '1'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,340 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:16,340 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1604): Closing 16362efeb81ea80068441d436c3937dd, disabling compactions & flushes 2023-07-12 15:40:16,341 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:16,341 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:16,341 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. after waiting 0 ms 2023-07-12 15:40:16,341 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:16,341 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:16,341 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:16,341 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:16,342 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1558): Region close journal for 16362efeb81ea80068441d436c3937dd: 2023-07-12 15:40:16,342 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1604): Closing 964fa8dd605f4f75805eb52796edffe1, disabling compactions & flushes 2023-07-12 15:40:16,342 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1604): Closing 7e64a90200e73af2749582231be39d70, disabling compactions & flushes 2023-07-12 15:40:16,343 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:16,343 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:16,343 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:16,343 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(7675): creating {ENCODED => b42cbcb78cb9d76bf763969942764e12, NAME => 'testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,343 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. after waiting 0 ms 2023-07-12 15:40:16,343 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:16,343 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:16,343 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. after waiting 0 ms 2023-07-12 15:40:16,343 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:16,343 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:16,343 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1558): Region close journal for 7e64a90200e73af2749582231be39d70: 2023-07-12 15:40:16,343 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:16,343 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1558): Region close journal for 964fa8dd605f4f75805eb52796edffe1: 2023-07-12 15:40:16,344 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 64ce4c9ba49998f908bf20dec4456a81, NAME => 'testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,344 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 6917e0806fe754e18b86baa9e5ba8962, NAME => 'testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=26 2023-07-12 15:40:16,762 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1604): Closing b42cbcb78cb9d76bf763969942764e12, disabling compactions & flushes 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1604): Closing 64ce4c9ba49998f908bf20dec4456a81, disabling compactions & flushes 2023-07-12 15:40:16,763 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:16,763 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. after waiting 0 ms 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1604): Closing 6917e0806fe754e18b86baa9e5ba8962, disabling compactions & flushes 2023-07-12 15:40:16,763 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:16,763 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. after waiting 0 ms 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1558): Region close journal for 64ce4c9ba49998f908bf20dec4456a81: 2023-07-12 15:40:16,764 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. after waiting 0 ms 2023-07-12 15:40:16,763 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:16,764 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:16,764 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:16,764 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:16,764 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1558): Region close journal for 6917e0806fe754e18b86baa9e5ba8962: 2023-07-12 15:40:16,764 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1558): Region close journal for b42cbcb78cb9d76bf763969942764e12: 2023-07-12 15:40:16,765 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6c78480cd12af27c2af33c2a96e94e5e, NAME => 'testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e.', STARTKEY => '7', ENDKEY => '9'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,765 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 1aeb1aec90691c00f122ca5f9a55e824, NAME => 'testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824.', STARTKEY => '6', ENDKEY => '7'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,765 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(7675): creating {ENCODED => e925615655bfa901ee0af32750a71b5d, NAME => 'testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d.', STARTKEY => '5', ENDKEY => '6'}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:16,852 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x43947ef2 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:16,856 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cef9256, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:16,858 DEBUG [hconnection-0x48935691-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:16,860 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:16,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=26 2023-07-12 15:40:17,174 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,174 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,174 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1604): Closing 1aeb1aec90691c00f122ca5f9a55e824, disabling compactions & flushes 2023-07-12 15:40:17,174 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,174 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:17,174 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1604): Closing e925615655bfa901ee0af32750a71b5d, disabling compactions & flushes 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:17,175 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1604): Closing 6c78480cd12af27c2af33c2a96e94e5e, disabling compactions & flushes 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. after waiting 0 ms 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. after waiting 0 ms 2023-07-12 15:40:17,175 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. after waiting 0 ms 2023-07-12 15:40:17,175 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:17,175 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:17,175 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1558): Region close journal for e925615655bfa901ee0af32750a71b5d: 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-0] regionserver.HRegion(1558): Region close journal for 6c78480cd12af27c2af33c2a96e94e5e: 2023-07-12 15:40:17,175 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-2] regionserver.HRegion(1558): Region close journal for 1aeb1aec90691c00f122ca5f9a55e824: 2023-07-12 15:40:17,176 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(7675): creating {ENCODED => aa169f7820d9c1d0085da2db0f07e85a, NAME => 'testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a.', STARTKEY => '9', ENDKEY => ''}, tableDescriptor='testTablesWithSnapshots4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:17,583 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,583 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1604): Closing aa169f7820d9c1d0085da2db0f07e85a, disabling compactions & flushes 2023-07-12 15:40:17,583 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:17,583 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:17,583 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. after waiting 0 ms 2023-07-12 15:40:17,583 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:17,583 INFO [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:17,583 DEBUG [RegionOpenAndInit-testTablesWithSnapshots4-pool-1] regionserver.HRegion(1558): Region close journal for aa169f7820d9c1d0085da2db0f07e85a: 2023-07-12 15:40:17,586 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=26, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testTablesWithSnapshots4 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:40:17,587 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd.","families":{"info":[{"qualifier":"regioninfo","vlen":59,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,588 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a.","families":{"info":[{"qualifier":"regioninfo","vlen":59,"tag":[],"timestamp":"1689176417587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176417587"}]},"ts":"1689176417587"} 2023-07-12 15:40:17,590 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 10 regions to meta. 2023-07-12 15:40:17,591 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=26, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testTablesWithSnapshots4 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:40:17,591 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTablesWithSnapshots4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176417591"}]},"ts":"1689176417591"} 2023-07-12 15:40:17,592 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testTablesWithSnapshots4, state=ENABLING in hbase:meta 2023-07-12 15:40:17,596 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=16362efeb81ea80068441d436c3937dd, ASSIGN}, {pid=28, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=7e64a90200e73af2749582231be39d70, ASSIGN}, {pid=29, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=964fa8dd605f4f75805eb52796edffe1, ASSIGN}, {pid=30, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=b42cbcb78cb9d76bf763969942764e12, ASSIGN}, {pid=31, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=64ce4c9ba49998f908bf20dec4456a81, ASSIGN}, {pid=32, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=6917e0806fe754e18b86baa9e5ba8962, ASSIGN}, {pid=33, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=e925615655bfa901ee0af32750a71b5d, ASSIGN}, {pid=34, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=1aeb1aec90691c00f122ca5f9a55e824, ASSIGN}, {pid=35, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=6c78480cd12af27c2af33c2a96e94e5e, ASSIGN}, {pid=36, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=aa169f7820d9c1d0085da2db0f07e85a, ASSIGN}] 2023-07-12 15:40:17,598 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=7e64a90200e73af2749582231be39d70, ASSIGN 2023-07-12 15:40:17,598 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=16362efeb81ea80068441d436c3937dd, ASSIGN 2023-07-12 15:40:17,598 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=964fa8dd605f4f75805eb52796edffe1, ASSIGN 2023-07-12 15:40:17,598 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=b42cbcb78cb9d76bf763969942764e12, ASSIGN 2023-07-12 15:40:17,599 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=7e64a90200e73af2749582231be39d70, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,599 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=27, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=16362efeb81ea80068441d436c3937dd, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,599 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=964fa8dd605f4f75805eb52796edffe1, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,599 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=b42cbcb78cb9d76bf763969942764e12, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,600 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=aa169f7820d9c1d0085da2db0f07e85a, ASSIGN 2023-07-12 15:40:17,600 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=35, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=6c78480cd12af27c2af33c2a96e94e5e, ASSIGN 2023-07-12 15:40:17,600 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=34, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=1aeb1aec90691c00f122ca5f9a55e824, ASSIGN 2023-07-12 15:40:17,600 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=33, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=e925615655bfa901ee0af32750a71b5d, ASSIGN 2023-07-12 15:40:17,600 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=6917e0806fe754e18b86baa9e5ba8962, ASSIGN 2023-07-12 15:40:17,601 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=aa169f7820d9c1d0085da2db0f07e85a, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,601 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=35, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=6c78480cd12af27c2af33c2a96e94e5e, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,601 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=33, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=e925615655bfa901ee0af32750a71b5d, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,601 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=34, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=1aeb1aec90691c00f122ca5f9a55e824, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,601 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=32, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=6917e0806fe754e18b86baa9e5ba8962, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,601 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=64ce4c9ba49998f908bf20dec4456a81, ASSIGN 2023-07-12 15:40:17,602 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=26, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=64ce4c9ba49998f908bf20dec4456a81, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:17,752 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=6917e0806fe754e18b86baa9e5ba8962, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,752 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=34 updating hbase:meta row=1aeb1aec90691c00f122ca5f9a55e824, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,752 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=16362efeb81ea80068441d436c3937dd, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,752 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=aa169f7820d9c1d0085da2db0f07e85a, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,753 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417752"}]},"ts":"1689176417752"} 2023-07-12 15:40:17,752 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=64ce4c9ba49998f908bf20dec4456a81, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,753 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a.","families":{"info":[{"qualifier":"regioninfo","vlen":59,"tag":[],"timestamp":"1689176417752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417752"}]},"ts":"1689176417752"} 2023-07-12 15:40:17,753 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417752"}]},"ts":"1689176417752"} 2023-07-12 15:40:17,753 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd.","families":{"info":[{"qualifier":"regioninfo","vlen":59,"tag":[],"timestamp":"1689176417752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417752"}]},"ts":"1689176417752"} 2023-07-12 15:40:17,753 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417752"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417752"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417752"}]},"ts":"1689176417752"} 2023-07-12 15:40:17,755 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=34, state=RUNNABLE; OpenRegionProcedure 1aeb1aec90691c00f122ca5f9a55e824, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,755 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=38, ppid=36, state=RUNNABLE; OpenRegionProcedure aa169f7820d9c1d0085da2db0f07e85a, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,756 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=31, state=RUNNABLE; OpenRegionProcedure 64ce4c9ba49998f908bf20dec4456a81, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=27, state=RUNNABLE; OpenRegionProcedure 16362efeb81ea80068441d436c3937dd, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=32, state=RUNNABLE; OpenRegionProcedure 6917e0806fe754e18b86baa9e5ba8962, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,757 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=7e64a90200e73af2749582231be39d70, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,757 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417757"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417757"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417757"}]},"ts":"1689176417757"} 2023-07-12 15:40:17,758 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=b42cbcb78cb9d76bf763969942764e12, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,758 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417758"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417758"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417758"}]},"ts":"1689176417758"} 2023-07-12 15:40:17,759 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=964fa8dd605f4f75805eb52796edffe1, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,759 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=28, state=RUNNABLE; OpenRegionProcedure 7e64a90200e73af2749582231be39d70, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,759 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417759"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417759"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417759"}]},"ts":"1689176417759"} 2023-07-12 15:40:17,759 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=33 updating hbase:meta row=e925615655bfa901ee0af32750a71b5d, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,759 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417759"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417759"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417759"}]},"ts":"1689176417759"} 2023-07-12 15:40:17,760 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=30, state=RUNNABLE; OpenRegionProcedure b42cbcb78cb9d76bf763969942764e12, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,760 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=35 updating hbase:meta row=6c78480cd12af27c2af33c2a96e94e5e, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,760 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=29, state=RUNNABLE; OpenRegionProcedure 964fa8dd605f4f75805eb52796edffe1, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,760 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417760"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176417760"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176417760"}]},"ts":"1689176417760"} 2023-07-12 15:40:17,761 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=33, state=RUNNABLE; OpenRegionProcedure e925615655bfa901ee0af32750a71b5d, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,762 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=35, state=RUNNABLE; OpenRegionProcedure 6c78480cd12af27c2af33c2a96e94e5e, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:17,804 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5a360753 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:17,814 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46947f23, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:17,816 DEBUG [hconnection-0x8292d1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:17,817 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:17,836 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,836 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,837 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,838 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,838 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:17,839 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,840 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:17,840 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:17,841 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,842 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:17,857 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,858 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,858 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,859 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,860 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:17,860 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,861 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:17,862 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:17,862 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,863 WARN [master/jenkins-hbase5:0.Chore.1] util.FSUtils$2(1259): Skipping region because it has no family dirs: hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:17,910 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:17,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1aeb1aec90691c00f122ca5f9a55e824, NAME => 'testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824.', STARTKEY => '6', ENDKEY => '7'} 2023-07-12 15:40:17,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,912 INFO [StoreOpener-1aeb1aec90691c00f122ca5f9a55e824-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,913 DEBUG [StoreOpener-1aeb1aec90691c00f122ca5f9a55e824-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1 2023-07-12 15:40:17,913 DEBUG [StoreOpener-1aeb1aec90691c00f122ca5f9a55e824-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1 2023-07-12 15:40:17,914 INFO [StoreOpener-1aeb1aec90691c00f122ca5f9a55e824-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1aeb1aec90691c00f122ca5f9a55e824 columnFamilyName f1 2023-07-12 15:40:17,914 INFO [StoreOpener-1aeb1aec90691c00f122ca5f9a55e824-1] regionserver.HStore(310): Store=1aeb1aec90691c00f122ca5f9a55e824/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:17,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:17,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:17,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 1aeb1aec90691c00f122ca5f9a55e824; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74205959, jitterRate=0.10575495660305023}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:17,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 1aeb1aec90691c00f122ca5f9a55e824: 2023-07-12 15:40:17,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824., pid=37, masterSystemTime=1689176417906 2023-07-12 15:40:17,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:17,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:17,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:17,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 16362efeb81ea80068441d436c3937dd, NAME => 'testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd.', STARTKEY => '', ENDKEY => '0'} 2023-07-12 15:40:17,923 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=34 updating hbase:meta row=1aeb1aec90691c00f122ca5f9a55e824, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,924 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417923"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176417923"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176417923"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176417923"}]},"ts":"1689176417923"} 2023-07-12 15:40:17,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,925 INFO [StoreOpener-16362efeb81ea80068441d436c3937dd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=34 2023-07-12 15:40:17,927 DEBUG [StoreOpener-16362efeb81ea80068441d436c3937dd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd/f1 2023-07-12 15:40:17,927 DEBUG [StoreOpener-16362efeb81ea80068441d436c3937dd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd/f1 2023-07-12 15:40:17,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=34, state=SUCCESS; OpenRegionProcedure 1aeb1aec90691c00f122ca5f9a55e824, server=jenkins-hbase5.apache.org,45089,1689176381369 in 170 msec 2023-07-12 15:40:17,927 INFO [StoreOpener-16362efeb81ea80068441d436c3937dd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 16362efeb81ea80068441d436c3937dd columnFamilyName f1 2023-07-12 15:40:17,928 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=1aeb1aec90691c00f122ca5f9a55e824, ASSIGN in 331 msec 2023-07-12 15:40:17,928 INFO [StoreOpener-16362efeb81ea80068441d436c3937dd-1] regionserver.HStore(310): Store=16362efeb81ea80068441d436c3937dd/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:17,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:17,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:17,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 16362efeb81ea80068441d436c3937dd; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63492854, jitterRate=-0.053882747888565063}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:17,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 16362efeb81ea80068441d436c3937dd: 2023-07-12 15:40:17,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd., pid=40, masterSystemTime=1689176417906 2023-07-12 15:40:17,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:17,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:17,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:17,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7e64a90200e73af2749582231be39d70, NAME => 'testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70.', STARTKEY => '0', ENDKEY => '1'} 2023-07-12 15:40:17,936 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=16362efeb81ea80068441d436c3937dd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,937 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd.","families":{"info":[{"qualifier":"regioninfo","vlen":59,"tag":[],"timestamp":"1689176417936"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176417936"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176417936"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176417936"}]},"ts":"1689176417936"} 2023-07-12 15:40:17,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,938 INFO [StoreOpener-7e64a90200e73af2749582231be39d70-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,939 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=27 2023-07-12 15:40:17,939 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=27, state=SUCCESS; OpenRegionProcedure 16362efeb81ea80068441d436c3937dd, server=jenkins-hbase5.apache.org,45089,1689176381369 in 182 msec 2023-07-12 15:40:17,940 DEBUG [StoreOpener-7e64a90200e73af2749582231be39d70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1 2023-07-12 15:40:17,940 DEBUG [StoreOpener-7e64a90200e73af2749582231be39d70-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1 2023-07-12 15:40:17,940 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=16362efeb81ea80068441d436c3937dd, ASSIGN in 343 msec 2023-07-12 15:40:17,941 INFO [StoreOpener-7e64a90200e73af2749582231be39d70-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7e64a90200e73af2749582231be39d70 columnFamilyName f1 2023-07-12 15:40:17,942 INFO [StoreOpener-7e64a90200e73af2749582231be39d70-1] regionserver.HStore(310): Store=7e64a90200e73af2749582231be39d70/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:17,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 7e64a90200e73af2749582231be39d70 2023-07-12 15:40:17,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:17,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 7e64a90200e73af2749582231be39d70; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74770625, jitterRate=0.11416913568973541}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:17,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 7e64a90200e73af2749582231be39d70: 2023-07-12 15:40:17,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70., pid=42, masterSystemTime=1689176417906 2023-07-12 15:40:17,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:17,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:17,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:17,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6917e0806fe754e18b86baa9e5ba8962, NAME => 'testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962.', STARTKEY => '4', ENDKEY => '5'} 2023-07-12 15:40:17,951 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=7e64a90200e73af2749582231be39d70, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,951 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417951"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176417951"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176417951"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176417951"}]},"ts":"1689176417951"} 2023-07-12 15:40:17,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,952 INFO [StoreOpener-6917e0806fe754e18b86baa9e5ba8962-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,954 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=28 2023-07-12 15:40:17,954 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=28, state=SUCCESS; OpenRegionProcedure 7e64a90200e73af2749582231be39d70, server=jenkins-hbase5.apache.org,45089,1689176381369 in 193 msec 2023-07-12 15:40:17,954 DEBUG [StoreOpener-6917e0806fe754e18b86baa9e5ba8962-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1 2023-07-12 15:40:17,954 DEBUG [StoreOpener-6917e0806fe754e18b86baa9e5ba8962-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1 2023-07-12 15:40:17,954 INFO [StoreOpener-6917e0806fe754e18b86baa9e5ba8962-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6917e0806fe754e18b86baa9e5ba8962 columnFamilyName f1 2023-07-12 15:40:17,955 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=7e64a90200e73af2749582231be39d70, ASSIGN in 358 msec 2023-07-12 15:40:17,955 INFO [StoreOpener-6917e0806fe754e18b86baa9e5ba8962-1] regionserver.HStore(310): Store=6917e0806fe754e18b86baa9e5ba8962/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:17,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:17,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:17,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 6917e0806fe754e18b86baa9e5ba8962; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64303828, jitterRate=-0.041798293590545654}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:17,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 6917e0806fe754e18b86baa9e5ba8962: 2023-07-12 15:40:17,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962., pid=41, masterSystemTime=1689176417906 2023-07-12 15:40:17,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:17,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:17,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:17,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 64ce4c9ba49998f908bf20dec4456a81, NAME => 'testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81.', STARTKEY => '3', ENDKEY => '4'} 2023-07-12 15:40:17,963 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=6917e0806fe754e18b86baa9e5ba8962, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,963 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417963"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176417963"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176417963"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176417963"}]},"ts":"1689176417963"} 2023-07-12 15:40:17,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,965 INFO [StoreOpener-64ce4c9ba49998f908bf20dec4456a81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,966 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=32 2023-07-12 15:40:17,966 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=32, state=SUCCESS; OpenRegionProcedure 6917e0806fe754e18b86baa9e5ba8962, server=jenkins-hbase5.apache.org,45089,1689176381369 in 208 msec 2023-07-12 15:40:17,967 DEBUG [StoreOpener-64ce4c9ba49998f908bf20dec4456a81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1 2023-07-12 15:40:17,967 DEBUG [StoreOpener-64ce4c9ba49998f908bf20dec4456a81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1 2023-07-12 15:40:17,967 INFO [StoreOpener-64ce4c9ba49998f908bf20dec4456a81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 64ce4c9ba49998f908bf20dec4456a81 columnFamilyName f1 2023-07-12 15:40:17,968 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=6917e0806fe754e18b86baa9e5ba8962, ASSIGN in 370 msec 2023-07-12 15:40:17,968 INFO [StoreOpener-64ce4c9ba49998f908bf20dec4456a81-1] regionserver.HStore(310): Store=64ce4c9ba49998f908bf20dec4456a81/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:17,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:17,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:17,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 64ce4c9ba49998f908bf20dec4456a81; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63080830, jitterRate=-0.06002238392829895}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:17,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 64ce4c9ba49998f908bf20dec4456a81: 2023-07-12 15:40:17,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81., pid=39, masterSystemTime=1689176417906 2023-07-12 15:40:17,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:17,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:17,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:17,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b42cbcb78cb9d76bf763969942764e12, NAME => 'testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12.', STARTKEY => '2', ENDKEY => '3'} 2023-07-12 15:40:17,976 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=64ce4c9ba49998f908bf20dec4456a81, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,976 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417976"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176417976"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176417976"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176417976"}]},"ts":"1689176417976"} 2023-07-12 15:40:17,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,978 INFO [StoreOpener-b42cbcb78cb9d76bf763969942764e12-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,979 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=31 2023-07-12 15:40:17,979 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=31, state=SUCCESS; OpenRegionProcedure 64ce4c9ba49998f908bf20dec4456a81, server=jenkins-hbase5.apache.org,45089,1689176381369 in 222 msec 2023-07-12 15:40:17,979 DEBUG [StoreOpener-b42cbcb78cb9d76bf763969942764e12-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1 2023-07-12 15:40:17,979 DEBUG [StoreOpener-b42cbcb78cb9d76bf763969942764e12-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1 2023-07-12 15:40:17,980 INFO [StoreOpener-b42cbcb78cb9d76bf763969942764e12-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b42cbcb78cb9d76bf763969942764e12 columnFamilyName f1 2023-07-12 15:40:17,980 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=64ce4c9ba49998f908bf20dec4456a81, ASSIGN in 383 msec 2023-07-12 15:40:17,981 INFO [StoreOpener-b42cbcb78cb9d76bf763969942764e12-1] regionserver.HStore(310): Store=b42cbcb78cb9d76bf763969942764e12/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:17,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:17,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:17,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened b42cbcb78cb9d76bf763969942764e12; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66002016, jitterRate=-0.01649332046508789}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:17,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for b42cbcb78cb9d76bf763969942764e12: 2023-07-12 15:40:17,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12., pid=43, masterSystemTime=1689176417906 2023-07-12 15:40:17,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:17,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:17,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:17,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e925615655bfa901ee0af32750a71b5d, NAME => 'testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d.', STARTKEY => '5', ENDKEY => '6'} 2023-07-12 15:40:17,989 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=b42cbcb78cb9d76bf763969942764e12, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:17,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:17,990 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176417989"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176417989"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176417989"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176417989"}]},"ts":"1689176417989"} 2023-07-12 15:40:17,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:17,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:17,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:17,991 INFO [StoreOpener-e925615655bfa901ee0af32750a71b5d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:17,993 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=30 2023-07-12 15:40:17,993 DEBUG [StoreOpener-e925615655bfa901ee0af32750a71b5d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1 2023-07-12 15:40:17,993 DEBUG [StoreOpener-e925615655bfa901ee0af32750a71b5d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1 2023-07-12 15:40:17,993 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=30, state=SUCCESS; OpenRegionProcedure b42cbcb78cb9d76bf763969942764e12, server=jenkins-hbase5.apache.org,45089,1689176381369 in 231 msec 2023-07-12 15:40:17,993 INFO [StoreOpener-e925615655bfa901ee0af32750a71b5d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e925615655bfa901ee0af32750a71b5d columnFamilyName f1 2023-07-12 15:40:17,994 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=b42cbcb78cb9d76bf763969942764e12, ASSIGN in 397 msec 2023-07-12 15:40:17,994 INFO [StoreOpener-e925615655bfa901ee0af32750a71b5d-1] regionserver.HStore(310): Store=e925615655bfa901ee0af32750a71b5d/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:17,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:17,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:17,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:18,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=26 2023-07-12 15:40:18,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:18,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened e925615655bfa901ee0af32750a71b5d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63407643, jitterRate=-0.055152490735054016}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:18,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for e925615655bfa901ee0af32750a71b5d: 2023-07-12 15:40:18,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d., pid=45, masterSystemTime=1689176417906 2023-07-12 15:40:18,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:18,003 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:18,003 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:18,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => aa169f7820d9c1d0085da2db0f07e85a, NAME => 'testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a.', STARTKEY => '9', ENDKEY => ''} 2023-07-12 15:40:18,003 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=33 updating hbase:meta row=e925615655bfa901ee0af32750a71b5d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:18,003 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176418003"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176418003"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176418003"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176418003"}]},"ts":"1689176418003"} 2023-07-12 15:40:18,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:18,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:18,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:18,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:18,005 INFO [StoreOpener-aa169f7820d9c1d0085da2db0f07e85a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:18,006 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=33 2023-07-12 15:40:18,006 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=33, state=SUCCESS; OpenRegionProcedure e925615655bfa901ee0af32750a71b5d, server=jenkins-hbase5.apache.org,45089,1689176381369 in 243 msec 2023-07-12 15:40:18,007 DEBUG [StoreOpener-aa169f7820d9c1d0085da2db0f07e85a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1 2023-07-12 15:40:18,007 DEBUG [StoreOpener-aa169f7820d9c1d0085da2db0f07e85a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1 2023-07-12 15:40:18,007 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=e925615655bfa901ee0af32750a71b5d, ASSIGN in 410 msec 2023-07-12 15:40:18,007 INFO [StoreOpener-aa169f7820d9c1d0085da2db0f07e85a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa169f7820d9c1d0085da2db0f07e85a columnFamilyName f1 2023-07-12 15:40:18,008 INFO [StoreOpener-aa169f7820d9c1d0085da2db0f07e85a-1] regionserver.HStore(310): Store=aa169f7820d9c1d0085da2db0f07e85a/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:18,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:18,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:18,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:18,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:18,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened aa169f7820d9c1d0085da2db0f07e85a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73143390, jitterRate=0.08992144465446472}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:18,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for aa169f7820d9c1d0085da2db0f07e85a: 2023-07-12 15:40:18,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a., pid=38, masterSystemTime=1689176417906 2023-07-12 15:40:18,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:18,017 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:18,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:18,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 964fa8dd605f4f75805eb52796edffe1, NAME => 'testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1.', STARTKEY => '1', ENDKEY => '2'} 2023-07-12 15:40:18,018 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=aa169f7820d9c1d0085da2db0f07e85a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:18,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:18,018 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a.","families":{"info":[{"qualifier":"regioninfo","vlen":59,"tag":[],"timestamp":"1689176418018"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176418018"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176418018"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176418018"}]},"ts":"1689176418018"} 2023-07-12 15:40:18,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:18,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:18,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:18,019 INFO [StoreOpener-964fa8dd605f4f75805eb52796edffe1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:18,021 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=38, resume processing ppid=36 2023-07-12 15:40:18,021 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; OpenRegionProcedure aa169f7820d9c1d0085da2db0f07e85a, server=jenkins-hbase5.apache.org,45089,1689176381369 in 264 msec 2023-07-12 15:40:18,021 DEBUG [StoreOpener-964fa8dd605f4f75805eb52796edffe1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1 2023-07-12 15:40:18,021 DEBUG [StoreOpener-964fa8dd605f4f75805eb52796edffe1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1 2023-07-12 15:40:18,022 INFO [StoreOpener-964fa8dd605f4f75805eb52796edffe1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 964fa8dd605f4f75805eb52796edffe1 columnFamilyName f1 2023-07-12 15:40:18,022 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=aa169f7820d9c1d0085da2db0f07e85a, ASSIGN in 425 msec 2023-07-12 15:40:18,022 INFO [StoreOpener-964fa8dd605f4f75805eb52796edffe1-1] regionserver.HStore(310): Store=964fa8dd605f4f75805eb52796edffe1/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:18,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:18,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:18,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:18,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:18,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 964fa8dd605f4f75805eb52796edffe1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70142614, jitterRate=0.04520639777183533}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:18,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 964fa8dd605f4f75805eb52796edffe1: 2023-07-12 15:40:18,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1., pid=44, masterSystemTime=1689176417906 2023-07-12 15:40:18,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:18,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:18,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:18,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6c78480cd12af27c2af33c2a96e94e5e, NAME => 'testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e.', STARTKEY => '7', ENDKEY => '9'} 2023-07-12 15:40:18,032 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=964fa8dd605f4f75805eb52796edffe1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:18,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesWithSnapshots4 6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:18,032 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176418032"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176418032"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176418032"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176418032"}]},"ts":"1689176418032"} 2023-07-12 15:40:18,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:18,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for 6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:18,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for 6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:18,034 INFO [StoreOpener-6c78480cd12af27c2af33c2a96e94e5e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:18,035 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=29 2023-07-12 15:40:18,035 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=29, state=SUCCESS; OpenRegionProcedure 964fa8dd605f4f75805eb52796edffe1, server=jenkins-hbase5.apache.org,45089,1689176381369 in 274 msec 2023-07-12 15:40:18,036 DEBUG [StoreOpener-6c78480cd12af27c2af33c2a96e94e5e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1 2023-07-12 15:40:18,036 DEBUG [StoreOpener-6c78480cd12af27c2af33c2a96e94e5e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1 2023-07-12 15:40:18,036 INFO [StoreOpener-6c78480cd12af27c2af33c2a96e94e5e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6c78480cd12af27c2af33c2a96e94e5e columnFamilyName f1 2023-07-12 15:40:18,036 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=964fa8dd605f4f75805eb52796edffe1, ASSIGN in 439 msec 2023-07-12 15:40:18,037 INFO [StoreOpener-6c78480cd12af27c2af33c2a96e94e5e-1] regionserver.HStore(310): Store=6c78480cd12af27c2af33c2a96e94e5e/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:18,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:18,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:18,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for 6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:18,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:18,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened 6c78480cd12af27c2af33c2a96e94e5e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72151295, jitterRate=0.07513807713985443}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:18,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for 6c78480cd12af27c2af33c2a96e94e5e: 2023-07-12 15:40:18,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e., pid=46, masterSystemTime=1689176417906 2023-07-12 15:40:18,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:18,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:18,045 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=35 updating hbase:meta row=6c78480cd12af27c2af33c2a96e94e5e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:18,045 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e.","families":{"info":[{"qualifier":"regioninfo","vlen":60,"tag":[],"timestamp":"1689176418045"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176418045"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176418045"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176418045"}]},"ts":"1689176418045"} 2023-07-12 15:40:18,048 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=35 2023-07-12 15:40:18,048 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=35, state=SUCCESS; OpenRegionProcedure 6c78480cd12af27c2af33c2a96e94e5e, server=jenkins-hbase5.apache.org,45089,1689176381369 in 284 msec 2023-07-12 15:40:18,049 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=35, resume processing ppid=26 2023-07-12 15:40:18,049 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=26, state=SUCCESS; TransitRegionStateProcedure table=testTablesWithSnapshots4, region=6c78480cd12af27c2af33c2a96e94e5e, ASSIGN in 452 msec 2023-07-12 15:40:18,050 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=26, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testTablesWithSnapshots4 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:40:18,050 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTablesWithSnapshots4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176418050"}]},"ts":"1689176418050"} 2023-07-12 15:40:18,051 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTablesWithSnapshots4, state=ENABLED in hbase:meta 2023-07-12 15:40:18,053 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=26, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testTablesWithSnapshots4 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:40:18,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, state=SUCCESS; CreateTableProcedure table=testTablesWithSnapshots4 in 2.1630 sec 2023-07-12 15:40:18,784 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTablesWithSnapshots4' 2023-07-12 15:40:18,805 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x23acde6c to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:18,810 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b0d7d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:18,811 DEBUG [hconnection-0x14c7ae61-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:18,814 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49332, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:18,900 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f3b7228 to 127.0.0.1:51623 2023-07-12 15:40:18,900 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,901 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4daf8ad5 to 127.0.0.1:51623 2023-07-12 15:40:18,901 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,901 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43947ef2 to 127.0.0.1:51623 2023-07-12 15:40:18,901 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,901 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a360753 to 127.0.0.1:51623 2023-07-12 15:40:18,901 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,902 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23acde6c to 127.0.0.1:51623 2023-07-12 15:40:18,902 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,902 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41c0446c to 127.0.0.1:51623 2023-07-12 15:40:18,902 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,902 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18da73fb to 127.0.0.1:51623 2023-07-12 15:40:18,902 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,902 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0cd75564 to 127.0.0.1:51623 2023-07-12 15:40:18,902 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,902 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5cbb07d0 to 127.0.0.1:51623 2023-07-12 15:40:18,902 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,903 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dee9e4f to 127.0.0.1:51623 2023-07-12 15:40:18,903 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,903 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x327e884f to 127.0.0.1:51623 2023-07-12 15:40:18,903 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,903 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d5e04d2 to 127.0.0.1:51623 2023-07-12 15:40:18,903 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,903 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x644a9ab7 to 127.0.0.1:51623 2023-07-12 15:40:18,904 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,904 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a14df5d to 127.0.0.1:51623 2023-07-12 15:40:18,904 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,904 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02cf6116 to 127.0.0.1:51623 2023-07-12 15:40:18,904 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,905 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a63e856 to 127.0.0.1:51623 2023-07-12 15:40:18,905 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,905 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76d911e8 to 127.0.0.1:51623 2023-07-12 15:40:18,905 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,905 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d37f35 to 127.0.0.1:51623 2023-07-12 15:40:18,905 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,906 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4daa16f6 to 127.0.0.1:51623 2023-07-12 15:40:18,906 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,906 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cb24a53 to 127.0.0.1:51623 2023-07-12 15:40:18,906 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,906 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d755e7c to 127.0.0.1:51623 2023-07-12 15:40:18,907 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,907 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13246c9f to 127.0.0.1:51623 2023-07-12 15:40:18,907 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,907 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c25527f to 127.0.0.1:51623 2023-07-12 15:40:18,907 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,907 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a4ac4d to 127.0.0.1:51623 2023-07-12 15:40:18,907 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,908 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210e6c9d to 127.0.0.1:51623 2023-07-12 15:40:18,909 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,909 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f2642f8 to 127.0.0.1:51623 2023-07-12 15:40:18,909 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,909 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22a0cb3c to 127.0.0.1:51623 2023-07-12 15:40:18,909 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,909 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x690f2901 to 127.0.0.1:51623 2023-07-12 15:40:18,909 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,909 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3879b067 to 127.0.0.1:51623 2023-07-12 15:40:18,909 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,909 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20e50169 to 127.0.0.1:51623 2023-07-12 15:40:18,909 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,909 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3153379f to 127.0.0.1:51623 2023-07-12 15:40:18,909 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:18,909 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31a704e3 to 127.0.0.1:51623 2023-07-12 15:40:18,909 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:19,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7e0a95e8 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:19,790 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f53d66d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:19,792 DEBUG [hconnection-0x6ea3de9d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:19,795 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:20,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=26 2023-07-12 15:40:20,002 INFO [Listener at localhost/38241] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testTablesWithSnapshots4, procId: 26 completed 2023-07-12 15:40:20,005 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(310): Writing first data set 2023-07-12 15:40:20,443 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 15:40:20,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:20,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testTablesWithSnapshots4 2023-07-12 15:40:20,450 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testTablesWithSnapshots4' 2023-07-12 15:40:20,450 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:20,451 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testTablesWithSnapshots4' starting 'acquire' 2023-07-12 15:40:20,451 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testTablesWithSnapshots4', kicking off acquire phase on members. 2023-07-12 15:40:20,451 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:20,451 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:20,452 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:20,452 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:20,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:20,453 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:20,453 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:20,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:20,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testTablesWithSnapshots4 2023-07-12 15:40:20,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testTablesWithSnapshots4 2023-07-12 15:40:20,456 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testTablesWithSnapshots4' with timeout 60000ms 2023-07-12 15:40:20,456 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:20,459 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testTablesWithSnapshots4' starting 'acquire' stage 2023-07-12 15:40:20,459 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:20,459 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. started... 2023-07-12 15:40:20,463 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b42cbcb78cb9d76bf763969942764e12 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:20,463 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 10 regions 2023-07-12 15:40:20,463 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:20,463 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:20,463 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. started... 2023-07-12 15:40:20,463 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:20,464 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2745): Flushing 7e64a90200e73af2749582231be39d70 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:20,464 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. started... 2023-07-12 15:40:20,465 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing aa169f7820d9c1d0085da2db0f07e85a 1/1 column families, dataSize=113.33 KB heapSize=144.04 KB 2023-07-12 15:40:20,506 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/.tmp/f1/83ebfcc786f64b08b873f7955371fc38 2023-07-12 15:40:20,516 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.33 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/.tmp/f1/67bebbbbc3ea4b81a180d0149480fd98 2023-07-12 15:40:20,517 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/.tmp/f1/83ebfcc786f64b08b873f7955371fc38 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/83ebfcc786f64b08b873f7955371fc38 2023-07-12 15:40:20,525 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/83ebfcc786f64b08b873f7955371fc38, entries=410, sequenceid=85, filesize=119.3 K 2023-07-12 15:40:20,526 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/.tmp/f1/67bebbbbc3ea4b81a180d0149480fd98 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/67bebbbbc3ea4b81a180d0149480fd98 2023-07-12 15:40:20,526 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for b42cbcb78cb9d76bf763969942764e12 in 63ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:20,532 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b42cbcb78cb9d76bf763969942764e12: 2023-07-12 15:40:20,532 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:20,532 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:20,532 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/10 local region flush tasks. 2023-07-12 15:40:20,532 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. started... 2023-07-12 15:40:20,533 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 964fa8dd605f4f75805eb52796edffe1 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:20,541 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/67bebbbbc3ea4b81a180d0149480fd98, entries=409, sequenceid=85, filesize=119.0 K 2023-07-12 15:40:20,543 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~113.33 KB/116045, heapSize ~144.02 KB/147480, currentSize=0 B/0 for aa169f7820d9c1d0085da2db0f07e85a in 78ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:20,543 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for aa169f7820d9c1d0085da2db0f07e85a: 2023-07-12 15:40:20,543 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:20,543 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 2/10 local region flush tasks. 2023-07-12 15:40:20,543 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:20,543 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. started... 2023-07-12 15:40:20,551 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for 16362efeb81ea80068441d436c3937dd: 2023-07-12 15:40:20,551 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:20,551 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:20,551 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 3/10 local region flush tasks. 2023-07-12 15:40:20,551 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. started... 2023-07-12 15:40:20,552 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing 6917e0806fe754e18b86baa9e5ba8962 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:20,555 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/.tmp/f1/a09b8ee0fae04691bcce0baeeec13876 2023-07-12 15:40:20,557 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/.tmp/f1/9ae9c618cb9046e1886b411877253f27 2023-07-12 15:40:20,572 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/.tmp/f1/a09b8ee0fae04691bcce0baeeec13876 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/a09b8ee0fae04691bcce0baeeec13876 2023-07-12 15:40:20,572 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/.tmp/f1/9ae9c618cb9046e1886b411877253f27 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/9ae9c618cb9046e1886b411877253f27 2023-07-12 15:40:20,580 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/9ae9c618cb9046e1886b411877253f27, entries=410, sequenceid=85, filesize=119.3 K 2023-07-12 15:40:20,580 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/a09b8ee0fae04691bcce0baeeec13876, entries=410, sequenceid=85, filesize=119.3 K 2023-07-12 15:40:20,581 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for 964fa8dd605f4f75805eb52796edffe1 in 48ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:20,581 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for 7e64a90200e73af2749582231be39d70 in 117ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:20,581 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 964fa8dd605f4f75805eb52796edffe1: 2023-07-12 15:40:20,581 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 7e64a90200e73af2749582231be39d70: 2023-07-12 15:40:20,581 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:20,581 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:20,581 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 4/10 local region flush tasks. 2023-07-12 15:40:20,581 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 5/10 local region flush tasks. 2023-07-12 15:40:20,581 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:20,581 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:20,581 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. started... 2023-07-12 15:40:20,581 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. started... 2023-07-12 15:40:20,588 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2745): Flushing e925615655bfa901ee0af32750a71b5d 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:20,588 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1aeb1aec90691c00f122ca5f9a55e824 1/1 column families, dataSize=113.33 KB heapSize=144.04 KB 2023-07-12 15:40:20,604 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.33 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/.tmp/f1/af3faa3511264e3ea8671918e7831092 2023-07-12 15:40:20,611 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/.tmp/f1/af3faa3511264e3ea8671918e7831092 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/af3faa3511264e3ea8671918e7831092 2023-07-12 15:40:20,616 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/af3faa3511264e3ea8671918e7831092, entries=409, sequenceid=85, filesize=119.0 K 2023-07-12 15:40:20,617 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~113.33 KB/116045, heapSize ~144.02 KB/147480, currentSize=0 B/0 for 1aeb1aec90691c00f122ca5f9a55e824 in 29ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:20,617 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1aeb1aec90691c00f122ca5f9a55e824: 2023-07-12 15:40:20,617 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:20,617 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:20,617 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 6/10 local region flush tasks. 2023-07-12 15:40:20,617 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. started... 2023-07-12 15:40:20,618 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6c78480cd12af27c2af33c2a96e94e5e 1/1 column families, dataSize=226.65 KB heapSize=287.83 KB 2023-07-12 15:40:20,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7d0c3ece to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:20,790 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dbb7c7a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:20,792 DEBUG [hconnection-0x6621925a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:20,793 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:20,877 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesWithSnapshots4 moved into observance of table space quota. 2023-07-12 15:40:20,976 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/.tmp/f1/4eda41aa2caa4a7a891f05f35428d9e5 2023-07-12 15:40:20,983 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/.tmp/f1/4eda41aa2caa4a7a891f05f35428d9e5 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/4eda41aa2caa4a7a891f05f35428d9e5 2023-07-12 15:40:20,991 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/4eda41aa2caa4a7a891f05f35428d9e5, entries=410, sequenceid=85, filesize=119.3 K 2023-07-12 15:40:20,992 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for 6917e0806fe754e18b86baa9e5ba8962 in 440ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:20,992 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for 6917e0806fe754e18b86baa9e5ba8962: 2023-07-12 15:40:20,992 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:20,992 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:20,992 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 7/10 local region flush tasks. 2023-07-12 15:40:20,992 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. started... 2023-07-12 15:40:20,993 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing 64ce4c9ba49998f908bf20dec4456a81 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:21,006 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/.tmp/f1/a19615574b964ff6bed94174a491efe1 2023-07-12 15:40:21,009 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/.tmp/f1/7d3929ef689049d5854cbbfb57174568 2023-07-12 15:40:21,017 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/.tmp/f1/a19615574b964ff6bed94174a491efe1 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/a19615574b964ff6bed94174a491efe1 2023-07-12 15:40:21,017 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/.tmp/f1/7d3929ef689049d5854cbbfb57174568 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/7d3929ef689049d5854cbbfb57174568 2023-07-12 15:40:21,023 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/7d3929ef689049d5854cbbfb57174568, entries=410, sequenceid=85, filesize=119.3 K 2023-07-12 15:40:21,024 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/a19615574b964ff6bed94174a491efe1, entries=410, sequenceid=85, filesize=119.3 K 2023-07-12 15:40:21,024 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for 64ce4c9ba49998f908bf20dec4456a81 in 31ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:21,024 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for 64ce4c9ba49998f908bf20dec4456a81: 2023-07-12 15:40:21,024 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:21,024 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 8/10 local region flush tasks. 2023-07-12 15:40:21,025 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for e925615655bfa901ee0af32750a71b5d in 437ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:21,025 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for e925615655bfa901ee0af32750a71b5d: 2023-07-12 15:40:21,025 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:21,025 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 9/10 local region flush tasks. 2023-07-12 15:40:21,027 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=226.65 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/.tmp/f1/00d4d3d956e14a319b96f7059b87bf92 2023-07-12 15:40:21,032 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/.tmp/f1/00d4d3d956e14a319b96f7059b87bf92 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/00d4d3d956e14a319b96f7059b87bf92 2023-07-12 15:40:21,037 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/00d4d3d956e14a319b96f7059b87bf92, entries=818, sequenceid=85, filesize=233.4 K 2023-07-12 15:40:21,038 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~226.65 KB/232090, heapSize ~287.81 KB/294720, currentSize=0 B/0 for 6c78480cd12af27c2af33c2a96e94e5e in 420ms, sequenceid=85, compaction requested=false 2023-07-12 15:40:21,038 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6c78480cd12af27c2af33c2a96e94e5e: 2023-07-12 15:40:21,038 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:21,038 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 10/10 local region flush tasks. 2023-07-12 15:40:21,038 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 10 local region flush tasks. 2023-07-12 15:40:21,038 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,038 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testTablesWithSnapshots4' locally acquired 2023-07-12 15:40:21,038 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testTablesWithSnapshots4) in zk 2023-07-12 15:40:21,040 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:21,040 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:21,041 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:21,041 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testTablesWithSnapshots4' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:21,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testTablesWithSnapshots4' on coordinator 2023-07-12 15:40:21,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8bc6e3[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:21,042 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testTablesWithSnapshots4' starting 'in-barrier' execution. 2023-07-12 15:40:21,042 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:21,044 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:21,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:21,044 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testTablesWithSnapshots4' received 'reached' from coordinator. 2023-07-12 15:40:21,044 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testTablesWithSnapshots4' locally completed 2023-07-12 15:40:21,044 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,044 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:21,044 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testTablesWithSnapshots4' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:21,047 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,047 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testTablesWithSnapshots4' has notified controller of completion 2023-07-12 15:40:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:21,047 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:21,047 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testTablesWithSnapshots4' completed. 2023-07-12 15:40:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testTablesWithSnapshots4' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:21,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testTablesWithSnapshots4', counting down latch. Waiting for 0 more 2023-07-12 15:40:21,050 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testTablesWithSnapshots4' execution completed 2023-07-12 15:40:21,050 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:21,050 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:21,050 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testTablesWithSnapshots4 2023-07-12 15:40:21,050 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testTablesWithSnapshots4including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:21,052 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:21,052 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:21,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:21,052 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:21,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:21,053 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:21,054 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,057 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,057 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:21,057 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:21,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:21,057 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:21,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testTablesWithSnapshots4' 2023-07-12 15:40:21,058 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:21,057 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:21,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:21,058 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,058 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:21,058 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:21,058 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:21,058 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testTablesWithSnapshots4'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:21,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:21,058 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:21,159 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:21,160 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 15:40:21,160 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(314): Creating snapshot 2023-07-12 15:40:21,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } 2023-07-12 15:40:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689176421161 (current time:1689176421161). 2023-07-12 15:40:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 15:40:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] zookeeper.ReadOnlyZKClient(139): Connect 0x597b1b3e to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:21,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75f7784e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:21,167 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49356, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:21,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x597b1b3e to 127.0.0.1:51623 2023-07-12 15:40:21,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 15:40:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } 2023-07-12 15:40:21,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=EXCLUSIVE 2023-07-12 15:40:21,171 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=47, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=EXCLUSIVE 2023-07-12 15:40:21,172 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=47, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=EXCLUSIVE; timeout=600000, timestamp=1689177021172 2023-07-12 15:40:21,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(653): Started snapshot: { ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } 2023-07-12 15:40:21,172 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot testTablesWithSnapshots4snap1 C_M_SNAPSHOT_TABLE on table testTablesWithSnapshots4 2023-07-12 15:40:21,173 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:21,173 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 15:40:21,173 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=47, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=EXCLUSIVE 2023-07-12 15:40:21,173 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=48, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=SHARED 2023-07-12 15:40:21,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=EXCLUSIVE in 3 msec 2023-07-12 15:40:21,175 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=48, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=SHARED 2023-07-12 15:40:21,175 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=48, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=SHARED; timeout=600000, timestamp=1689177021175 2023-07-12 15:40:21,273 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:21,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } is done 2023-07-12 15:40:21,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:21,275 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 15:40:21,475 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } is done 2023-07-12 15:40:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:21,476 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 15:40:21,585 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testTablesWithSnapshots4snap1 2023-07-12 15:40:21,585 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testTablesWithSnapshots4snap1' 2023-07-12 15:40:21,585 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 15:40:21,585 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testTablesWithSnapshots4snap1' starting 'acquire' 2023-07-12 15:40:21,585 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testTablesWithSnapshots4snap1', kicking off acquire phase on members. 2023-07-12 15:40:21,586 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,586 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,587 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,587 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 15:40:21,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 15:40:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:40:21,587 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,588 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 74 2023-07-12 15:40:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,588 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testTablesWithSnapshots4snap1 from table testTablesWithSnapshots4 type FLUSH 2023-07-12 15:40:21,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testTablesWithSnapshots4snap1 2023-07-12 15:40:21,589 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testTablesWithSnapshots4snap1' with timeout 300000ms 2023-07-12 15:40:21,589 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 15:40:21,589 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testTablesWithSnapshots4snap1' starting 'acquire' stage 2023-07-12 15:40:21,589 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testTablesWithSnapshots4snap1' locally acquired 2023-07-12 15:40:21,589 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testTablesWithSnapshots4snap1) in zk 2023-07-12 15:40:21,593 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,593 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:21,593 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,593 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testTablesWithSnapshots4snap1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4snap1 2023-07-12 15:40:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:21,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testTablesWithSnapshots4snap1' on coordinator 2023-07-12 15:40:21,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31331d7a[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:21,595 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testTablesWithSnapshots4snap1' starting 'in-barrier' execution. 2023-07-12 15:40:21,595 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,596 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1 2023-07-12 15:40:21,596 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testTablesWithSnapshots4snap1' received 'reached' from coordinator. 2023-07-12 15:40:21,596 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:21,597 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. started... 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:21,598 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 10 regions 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for b42cbcb78cb9d76bf763969942764e12: 2023-07-12 15:40:21,598 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. started... 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. started... 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] regionserver.HRegion(2446): Flush status journal for 7e64a90200e73af2749582231be39d70: 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] regionserver.HRegion(2446): Flush status journal for aa169f7820d9c1d0085da2db0f07e85a: 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/83ebfcc786f64b08b873f7955371fc38] hfiles 2023-07-12 15:40:21,598 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/83ebfcc786f64b08b873f7955371fc38 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:21,599 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:21,599 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:21,599 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:21,599 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:21,599 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/a09b8ee0fae04691bcce0baeeec13876] hfiles 2023-07-12 15:40:21,599 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/67bebbbbc3ea4b81a180d0149480fd98] hfiles 2023-07-12 15:40:21,599 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/a09b8ee0fae04691bcce0baeeec13876 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:21,599 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/67bebbbbc3ea4b81a180d0149480fd98 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:21,777 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } is done 2023-07-12 15:40:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:21,778 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 15:40:21,840 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesWithSnapshots4 moved into observance of table space quota. 2023-07-12 15:40:21,843 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x72e3d45b to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:21,847 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77aaadac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:21,855 DEBUG [hconnection-0x19ea6a34-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:21,858 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:22,013 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. completed. 2023-07-12 15:40:22,013 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:22,013 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:22,013 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/10 local region snapshots. 2023-07-12 15:40:22,013 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. started... 2023-07-12 15:40:22,013 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] regionserver.HRegion(2446): Flush status journal for 964fa8dd605f4f75805eb52796edffe1: 2023-07-12 15:40:22,013 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,013 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:22,014 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/9ae9c618cb9046e1886b411877253f27] hfiles 2023-07-12 15:40:22,014 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/9ae9c618cb9046e1886b411877253f27 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,015 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. completed. 2023-07-12 15:40:22,015 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:22,015 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:22,015 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. started... 2023-07-12 15:40:22,015 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. completed. 2023-07-12 15:40:22,015 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 16362efeb81ea80068441d436c3937dd: 2023-07-12 15:40:22,015 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:22,015 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. started... 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] regionserver.HRegion(2446): Flush status journal for 6917e0806fe754e18b86baa9e5ba8962: 2023-07-12 15:40:22,016 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 2/10 local region snapshots. 2023-07-12 15:40:22,016 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 3/10 local region snapshots. 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/4eda41aa2caa4a7a891f05f35428d9e5] hfiles 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/4eda41aa2caa4a7a891f05f35428d9e5 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:22,016 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [] hfiles 2023-07-12 15:40:22,278 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:22,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } is done 2023-07-12 15:40:22,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:22,280 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 15:40:22,433 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. completed. 2023-07-12 15:40:22,433 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:22,433 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:22,433 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. started... 2023-07-12 15:40:22,433 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] regionserver.HRegion(2446): Flush status journal for e925615655bfa901ee0af32750a71b5d: 2023-07-12 15:40:22,433 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/a19615574b964ff6bed94174a491efe1] hfiles 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/a19615574b964ff6bed94174a491efe1 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,434 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 4/10 local region snapshots. 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. completed. 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. completed. 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. started... 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for 1aeb1aec90691c00f122ca5f9a55e824: 2023-07-12 15:40:22,434 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. started... 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/af3faa3511264e3ea8671918e7831092] hfiles 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/af3faa3511264e3ea8671918e7831092 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] regionserver.HRegion(2446): Flush status journal for 6c78480cd12af27c2af33c2a96e94e5e: 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/00d4d3d956e14a319b96f7059b87bf92] hfiles 2023-07-12 15:40:22,435 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/00d4d3d956e14a319b96f7059b87bf92 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,435 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 5/10 local region snapshots. 2023-07-12 15:40:22,435 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 6/10 local region snapshots. 2023-07-12 15:40:22,799 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesWithSnapshots4 moved into observance of table space quota. 2023-07-12 15:40:22,802 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4e3de1d0 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:22,806 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59b698eb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:22,807 DEBUG [hconnection-0x4dad2ca2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:22,808 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:22,861 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. completed. 2023-07-12 15:40:22,861 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:22,861 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:22,861 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 7/10 local region snapshots. 2023-07-12 15:40:22,861 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. started... 2023-07-12 15:40:22,862 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] regionserver.HRegion(2446): Flush status journal for 64ce4c9ba49998f908bf20dec4456a81: 2023-07-12 15:40:22,862 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(238): Storing 'testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81.' region-info for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,862 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:22,862 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/7d3929ef689049d5854cbbfb57174568] hfiles 2023-07-12 15:40:22,862 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/7d3929ef689049d5854cbbfb57174568 for snapshot=testTablesWithSnapshots4snap1 2023-07-12 15:40:22,862 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. completed. 2023-07-12 15:40:22,862 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:22,863 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 8/10 local region snapshots. 2023-07-12 15:40:22,867 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. completed. 2023-07-12 15:40:22,867 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:22,867 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 9/10 local region snapshots. 2023-07-12 15:40:23,274 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. completed. 2023-07-12 15:40:23,274 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-2] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:23,274 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 10/10 local region snapshots. 2023-07-12 15:40:23,274 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 10 local region snapshots. 2023-07-12 15:40:23,274 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,274 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testTablesWithSnapshots4snap1' locally completed 2023-07-12 15:40:23,274 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testTablesWithSnapshots4snap1' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:23,276 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,276 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testTablesWithSnapshots4snap1' has notified controller of completion 2023-07-12 15:40:23,276 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:23,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:23,276 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testTablesWithSnapshots4snap1' completed. 2023-07-12 15:40:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4snap1 2023-07-12 15:40:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4snap1 2023-07-12 15:40:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testTablesWithSnapshots4snap1' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testTablesWithSnapshots4snap1', counting down latch. Waiting for 0 more 2023-07-12 15:40:23,280 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testTablesWithSnapshots4snap1' execution completed 2023-07-12 15:40:23,280 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:23,280 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:23,280 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testTablesWithSnapshots4snap1 2023-07-12 15:40:23,280 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testTablesWithSnapshots4snap1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 15:40:23,280 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:23,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } is done 2023-07-12 15:40:23,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:23,281 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#6) Sleeping: 2000ms while waiting for snapshot completion. 2023-07-12 15:40:23,281 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,282 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 15:40:23,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,282 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:23,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 15:40:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:40:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:23,282 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4snap1 2023-07-12 15:40:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4snap1 2023-07-12 15:40:23,284 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4snap1 2023-07-12 15:40:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,286 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 15:40:23,286 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 15:40:23,286 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,286 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 15:40:23,287 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,287 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testTablesWithSnapshots4snap1 2023-07-12 15:40:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:40:23,287 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testTablesWithSnapshots4snap1 2023-07-12 15:40:23,287 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:23,287 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:23,287 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,288 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,288 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 15:40:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:40:23,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/testTablesWithSnapshots4snap1 because node does not exist (not an error) 2023-07-12 15:40:23,291 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,448 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 15:40:23,726 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testTablesWithSnapshots4snap1 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,796 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1972b744 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:23,800 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7af263c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:23,802 DEBUG [hconnection-0x5384c345-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:23,803 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:23,861 DEBUG [master/jenkins-hbase5:0.Chore.1] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/testTablesWithSnapshots4snap1 2023-07-12 15:40:23,861 ERROR [master/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(166): Caught error java.lang.NullPointerException at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.getStoreFilesFromSnapshot(FileArchiverNotifierImpl.java:488) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeSnapshotSizes(FileArchiverNotifierImpl.java:404) at org.apache.hadoop.hbase.quotas.FileArchiverNotifierImpl.computeAndStoreSnapshotSizes(FileArchiverNotifierImpl.java:334) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.computeSnapshotSizes(SnapshotQuotaObserverChore.java:225) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:118) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:40:24,544 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testTablesWithSnapshots4snap1 of table testTablesWithSnapshots4 completed 2023-07-12 15:40:24,544 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testTablesWithSnapshots4snap1 2023-07-12 15:40:24,545 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testTablesWithSnapshots4snap1 2023-07-12 15:40:24,545 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot testTablesWithSnapshots4snap1 C_M_SNAPSHOT_TABLE on table testTablesWithSnapshots4 at 1689176421172Consolidate snapshot: testTablesWithSnapshots4snap1 at 1689176423287 (+2115 ms)Loading Region manifests for testTablesWithSnapshots4snap1 at 1689176423287Writing data manifest for testTablesWithSnapshots4snap1 at 1689176423306 (+19 ms)Verifying snapshot: testTablesWithSnapshots4snap1 at 1689176423716 (+410 ms)Snapshot testTablesWithSnapshots4snap1 of table testTablesWithSnapshots4 completed at 1689176424544 (+828 ms) 2023-07-12 15:40:24,547 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=48, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=SHARED 2023-07-12 15:40:24,548 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesWithSnapshots4, type=SHARED in 3.3740 sec 2023-07-12 15:40:24,788 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x5e301388 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:24,791 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ba2bd3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:24,793 DEBUG [hconnection-0x65af4c7b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:24,795 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49394, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:25,281 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:25,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 } is done 2023-07-12 15:40:25,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(401): Snapshot '{ ss=testTablesWithSnapshots4snap1 table=testTablesWithSnapshots4 type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 15:40:25,283 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(317): Writing second data set 2023-07-12 15:40:25,642 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 15:40:25,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:25,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure testTablesWithSnapshots4 was in running list but was completed. Accepting new attempt. 2023-07-12 15:40:25,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testTablesWithSnapshots4 2023-07-12 15:40:25,647 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testTablesWithSnapshots4' 2023-07-12 15:40:25,647 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:25,648 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testTablesWithSnapshots4' starting 'acquire' 2023-07-12 15:40:25,648 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testTablesWithSnapshots4', kicking off acquire phase on members. 2023-07-12 15:40:25,648 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:25,649 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:25,650 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:25,650 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:25,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:25,650 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:25,650 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:25,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:25,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testTablesWithSnapshots4 2023-07-12 15:40:25,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testTablesWithSnapshots4 is still present, removing 2023-07-12 15:40:25,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testTablesWithSnapshots4 2023-07-12 15:40:25,652 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testTablesWithSnapshots4' with timeout 60000ms 2023-07-12 15:40:25,652 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:25,652 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testTablesWithSnapshots4' starting 'acquire' stage 2023-07-12 15:40:25,652 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:25,652 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. started... 2023-07-12 15:40:25,652 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:25,652 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. started... 2023-07-12 15:40:25,653 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b42cbcb78cb9d76bf763969942764e12 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:25,653 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2745): Flushing 7e64a90200e73af2749582231be39d70 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:25,653 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 10 regions 2023-07-12 15:40:25,653 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:25,653 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. started... 2023-07-12 15:40:25,653 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:25,653 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing aa169f7820d9c1d0085da2db0f07e85a 1/1 column families, dataSize=113.33 KB heapSize=144.04 KB 2023-07-12 15:40:25,808 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x084dae70 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:25,811 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78c5b1cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:25,813 DEBUG [hconnection-0x3feb9e01-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:25,814 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:49404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:26,066 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/.tmp/f1/d4eccc47ff9246bf9de59513ae282a9d 2023-07-12 15:40:26,066 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/.tmp/f1/98f63fba38c1438b86bc9497933a78fa 2023-07-12 15:40:26,069 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.33 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/.tmp/f1/1065446dd9774726ba9ed61333fcbca3 2023-07-12 15:40:26,074 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/.tmp/f1/d4eccc47ff9246bf9de59513ae282a9d as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/d4eccc47ff9246bf9de59513ae282a9d 2023-07-12 15:40:26,074 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/.tmp/f1/98f63fba38c1438b86bc9497933a78fa as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/98f63fba38c1438b86bc9497933a78fa 2023-07-12 15:40:26,075 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/.tmp/f1/1065446dd9774726ba9ed61333fcbca3 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/1065446dd9774726ba9ed61333fcbca3 2023-07-12 15:40:26,082 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/d4eccc47ff9246bf9de59513ae282a9d, entries=410, sequenceid=170, filesize=119.5 K 2023-07-12 15:40:26,082 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/1065446dd9774726ba9ed61333fcbca3, entries=409, sequenceid=170, filesize=119.2 K 2023-07-12 15:40:26,082 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/98f63fba38c1438b86bc9497933a78fa, entries=410, sequenceid=170, filesize=119.5 K 2023-07-12 15:40:26,083 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~113.33 KB/116045, heapSize ~144.02 KB/147480, currentSize=0 B/0 for aa169f7820d9c1d0085da2db0f07e85a in 430ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,083 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for 7e64a90200e73af2749582231be39d70 in 430ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for aa169f7820d9c1d0085da2db0f07e85a: 2023-07-12 15:40:26,083 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for b42cbcb78cb9d76bf763969942764e12 in 431ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:26,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 7e64a90200e73af2749582231be39d70: 2023-07-12 15:40:26,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:26,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:26,083 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/10 local region flush tasks. 2023-07-12 15:40:26,083 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 2/10 local region flush tasks. 2023-07-12 15:40:26,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b42cbcb78cb9d76bf763969942764e12: 2023-07-12 15:40:26,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:26,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. started... 2023-07-12 15:40:26,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. started... 2023-07-12 15:40:26,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:26,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 16362efeb81ea80068441d436c3937dd: 2023-07-12 15:40:26,084 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing 964fa8dd605f4f75805eb52796edffe1 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:26,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:26,084 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 3/10 local region flush tasks. 2023-07-12 15:40:26,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. started... 2023-07-12 15:40:26,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:26,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:26,084 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 4/10 local region flush tasks. 2023-07-12 15:40:26,084 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6917e0806fe754e18b86baa9e5ba8962 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:26,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. started... 2023-07-12 15:40:26,085 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2745): Flushing e925615655bfa901ee0af32750a71b5d 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:26,498 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/.tmp/f1/5cfbaa27ae7b40f99490616ba3d3b7e0 2023-07-12 15:40:26,499 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/.tmp/f1/4233047df0154697ad99bf4f3c3311b4 2023-07-12 15:40:26,499 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/.tmp/f1/03af4ad782c14887b377de6535a570ae 2023-07-12 15:40:26,506 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/.tmp/f1/5cfbaa27ae7b40f99490616ba3d3b7e0 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/5cfbaa27ae7b40f99490616ba3d3b7e0 2023-07-12 15:40:26,510 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/.tmp/f1/4233047df0154697ad99bf4f3c3311b4 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/4233047df0154697ad99bf4f3c3311b4 2023-07-12 15:40:26,510 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/.tmp/f1/03af4ad782c14887b377de6535a570ae as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/03af4ad782c14887b377de6535a570ae 2023-07-12 15:40:26,512 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/5cfbaa27ae7b40f99490616ba3d3b7e0, entries=410, sequenceid=170, filesize=119.5 K 2023-07-12 15:40:26,513 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for e925615655bfa901ee0af32750a71b5d in 428ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,513 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for e925615655bfa901ee0af32750a71b5d: 2023-07-12 15:40:26,513 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:26,513 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:26,513 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. started... 2023-07-12 15:40:26,514 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2745): Flushing 1aeb1aec90691c00f122ca5f9a55e824 1/1 column families, dataSize=113.33 KB heapSize=144.04 KB 2023-07-12 15:40:26,514 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 5/10 local region flush tasks. 2023-07-12 15:40:26,517 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/03af4ad782c14887b377de6535a570ae, entries=410, sequenceid=170, filesize=119.5 K 2023-07-12 15:40:26,517 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/4233047df0154697ad99bf4f3c3311b4, entries=410, sequenceid=170, filesize=119.5 K 2023-07-12 15:40:26,518 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for 6917e0806fe754e18b86baa9e5ba8962 in 434ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,518 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for 964fa8dd605f4f75805eb52796edffe1 in 434ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,518 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6917e0806fe754e18b86baa9e5ba8962: 2023-07-12 15:40:26,518 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for 964fa8dd605f4f75805eb52796edffe1: 2023-07-12 15:40:26,518 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:26,518 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:26,518 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 6/10 local region flush tasks. 2023-07-12 15:40:26,518 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:26,518 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:26,518 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. started... 2023-07-12 15:40:26,518 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. started... 2023-07-12 15:40:26,518 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 7/10 local region flush tasks. 2023-07-12 15:40:26,518 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2745): Flushing 64ce4c9ba49998f908bf20dec4456a81 1/1 column families, dataSize=113.60 KB heapSize=144.39 KB 2023-07-12 15:40:26,518 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6c78480cd12af27c2af33c2a96e94e5e 1/1 column families, dataSize=226.65 KB heapSize=287.83 KB 2023-07-12 15:40:26,838 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x12b3e80c to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:26,842 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f321c4b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:26,843 DEBUG [hconnection-0x63d909e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:26,845 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:26,921 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesWithSnapshots4 moved into observance of table space quota. 2023-07-12 15:40:26,924 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.33 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/.tmp/f1/5f1c095d3dbb4b9aa5278baea8dc9376 2023-07-12 15:40:26,929 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/.tmp/f1/5f1c095d3dbb4b9aa5278baea8dc9376 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/5f1c095d3dbb4b9aa5278baea8dc9376 2023-07-12 15:40:26,930 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=226.65 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/.tmp/f1/702a9d26e1a64fb4bd6f1367f4680066 2023-07-12 15:40:26,933 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/5f1c095d3dbb4b9aa5278baea8dc9376, entries=409, sequenceid=170, filesize=119.2 K 2023-07-12 15:40:26,934 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=113.60 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/.tmp/f1/9dd856192f854cc8b3ce36bc5960ae82 2023-07-12 15:40:26,934 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2948): Finished flush of dataSize ~113.33 KB/116045, heapSize ~144.02 KB/147480, currentSize=0 B/0 for 1aeb1aec90691c00f122ca5f9a55e824 in 421ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,934 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 1aeb1aec90691c00f122ca5f9a55e824: 2023-07-12 15:40:26,934 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:26,935 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 8/10 local region flush tasks. 2023-07-12 15:40:26,936 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/.tmp/f1/702a9d26e1a64fb4bd6f1367f4680066 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/702a9d26e1a64fb4bd6f1367f4680066 2023-07-12 15:40:26,939 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/.tmp/f1/9dd856192f854cc8b3ce36bc5960ae82 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/9dd856192f854cc8b3ce36bc5960ae82 2023-07-12 15:40:26,941 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/702a9d26e1a64fb4bd6f1367f4680066, entries=818, sequenceid=170, filesize=233.8 K 2023-07-12 15:40:26,941 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~226.65 KB/232090, heapSize ~287.81 KB/294720, currentSize=0 B/0 for 6c78480cd12af27c2af33c2a96e94e5e in 423ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,941 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6c78480cd12af27c2af33c2a96e94e5e: 2023-07-12 15:40:26,941 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:26,941 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 9/10 local region flush tasks. 2023-07-12 15:40:26,943 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/9dd856192f854cc8b3ce36bc5960ae82, entries=410, sequenceid=170, filesize=119.5 K 2023-07-12 15:40:26,944 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2948): Finished flush of dataSize ~113.60 KB/116329, heapSize ~144.38 KB/147840, currentSize=0 B/0 for 64ce4c9ba49998f908bf20dec4456a81 in 426ms, sequenceid=170, compaction requested=false 2023-07-12 15:40:26,944 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for 64ce4c9ba49998f908bf20dec4456a81: 2023-07-12 15:40:26,944 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:26,944 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 10/10 local region flush tasks. 2023-07-12 15:40:26,944 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 10 local region flush tasks. 2023-07-12 15:40:26,944 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,944 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testTablesWithSnapshots4' locally acquired 2023-07-12 15:40:26,944 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testTablesWithSnapshots4) in zk 2023-07-12 15:40:26,946 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,946 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:26,946 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:26,946 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testTablesWithSnapshots4' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:26,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testTablesWithSnapshots4' on coordinator 2023-07-12 15:40:26,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51b12a21[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:26,948 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testTablesWithSnapshots4' starting 'in-barrier' execution. 2023-07-12 15:40:26,948 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:26,951 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:26,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:26,951 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testTablesWithSnapshots4' received 'reached' from coordinator. 2023-07-12 15:40:26,951 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testTablesWithSnapshots4' locally completed 2023-07-12 15:40:26,951 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testTablesWithSnapshots4' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:26,951 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,951 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:26,953 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testTablesWithSnapshots4' has notified controller of completion 2023-07-12 15:40:26,953 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,953 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:26,953 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testTablesWithSnapshots4' completed. 2023-07-12 15:40:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testTablesWithSnapshots4' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:26,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testTablesWithSnapshots4', counting down latch. Waiting for 0 more 2023-07-12 15:40:26,957 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testTablesWithSnapshots4' execution completed 2023-07-12 15:40:26,957 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:26,957 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:26,957 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testTablesWithSnapshots4 2023-07-12 15:40:26,957 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testTablesWithSnapshots4including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:26,962 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:26,962 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:26,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:26,962 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:26,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:26,963 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:26,965 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,969 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,969 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:26,969 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:26,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:26,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testTablesWithSnapshots4' 2023-07-12 15:40:26,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:26,969 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:26,969 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:26,969 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:26,970 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testTablesWithSnapshots4'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:26,970 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:26,970 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:26,970 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:26,970 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:26,970 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:26,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testTablesWithSnapshots4 because node does not exist (not an error) 2023-07-12 15:40:26,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:27,070 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:27,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:27,071 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 15:40:27,072 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(321): Flushing and major compacting table 2023-07-12 15:40:27,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:27,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure testTablesWithSnapshots4 was in running list but was completed. Accepting new attempt. 2023-07-12 15:40:27,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testTablesWithSnapshots4 2023-07-12 15:40:27,077 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testTablesWithSnapshots4' 2023-07-12 15:40:27,077 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:27,077 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testTablesWithSnapshots4' starting 'acquire' 2023-07-12 15:40:27,077 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testTablesWithSnapshots4', kicking off acquire phase on members. 2023-07-12 15:40:27,078 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,078 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:27,079 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,079 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:27,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:27,080 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,080 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:27,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:27,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testTablesWithSnapshots4 2023-07-12 15:40:27,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testTablesWithSnapshots4 is still present, removing 2023-07-12 15:40:27,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testTablesWithSnapshots4 2023-07-12 15:40:27,081 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testTablesWithSnapshots4' with timeout 60000ms 2023-07-12 15:40:27,081 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:27,082 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testTablesWithSnapshots4' starting 'acquire' stage 2023-07-12 15:40:27,082 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:27,082 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. started... 2023-07-12 15:40:27,082 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:27,082 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:27,082 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. started... 2023-07-12 15:40:27,082 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b42cbcb78cb9d76bf763969942764e12: 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for aa169f7820d9c1d0085da2db0f07e85a: 2023-07-12 15:40:27,082 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. started... 2023-07-12 15:40:27,082 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 10 regions 2023-07-12 15:40:27,083 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. started... 2023-07-12 15:40:27,083 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/10 local region flush tasks. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 16362efeb81ea80068441d436c3937dd: 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 7e64a90200e73af2749582231be39d70: 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. started... 2023-07-12 15:40:27,083 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 2/10 local region flush tasks. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6917e0806fe754e18b86baa9e5ba8962: 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. started... 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:27,083 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 3/10 local region flush tasks. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for 964fa8dd605f4f75805eb52796edffe1: 2023-07-12 15:40:27,084 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 4/10 local region flush tasks. 2023-07-12 15:40:27,083 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:27,084 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 5/10 local region flush tasks. 2023-07-12 15:40:27,084 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 6/10 local region flush tasks. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. started... 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e925615655bfa901ee0af32750a71b5d: 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. started... 2023-07-12 15:40:27,084 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 7/10 local region flush tasks. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] regionserver.HRegion(2446): Flush status journal for 6c78480cd12af27c2af33c2a96e94e5e: 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-2] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:27,084 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 8/10 local region flush tasks. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. started... 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. started... 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] regionserver.HRegion(2446): Flush status journal for 1aeb1aec90691c00f122ca5f9a55e824: 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 64ce4c9ba49998f908bf20dec4456a81: 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:27,084 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-1] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:27,085 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 9/10 local region flush tasks. 2023-07-12 15:40:27,085 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 10/10 local region flush tasks. 2023-07-12 15:40:27,085 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 10 local region flush tasks. 2023-07-12 15:40:27,085 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,085 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testTablesWithSnapshots4' locally acquired 2023-07-12 15:40:27,085 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testTablesWithSnapshots4) in zk 2023-07-12 15:40:27,087 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,087 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:27,087 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:27,087 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testTablesWithSnapshots4' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:27,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testTablesWithSnapshots4' on coordinator 2023-07-12 15:40:27,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3027cdb2[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:27,089 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testTablesWithSnapshots4' starting 'in-barrier' execution. 2023-07-12 15:40:27,089 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:27,090 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:27,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:27,090 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testTablesWithSnapshots4' received 'reached' from coordinator. 2023-07-12 15:40:27,090 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testTablesWithSnapshots4' locally completed 2023-07-12 15:40:27,090 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testTablesWithSnapshots4' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:27,090 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,090 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:27,093 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testTablesWithSnapshots4' has notified controller of completion 2023-07-12 15:40:27,093 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,093 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:27,093 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testTablesWithSnapshots4' completed. 2023-07-12 15:40:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testTablesWithSnapshots4' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:27,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testTablesWithSnapshots4', counting down latch. Waiting for 0 more 2023-07-12 15:40:27,096 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testTablesWithSnapshots4' execution completed 2023-07-12 15:40:27,096 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:27,096 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:27,096 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testTablesWithSnapshots4 2023-07-12 15:40:27,096 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testTablesWithSnapshots4including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:27,098 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,098 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:27,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,098 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:27,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,098 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,100 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesWithSnapshots4 2023-07-12 15:40:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,101 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,102 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:27,102 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:27,102 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:27,102 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesWithSnapshots4 2023-07-12 15:40:27,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:27,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testTablesWithSnapshots4' 2023-07-12 15:40:27,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:27,102 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:27,102 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:27,102 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:27,103 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesWithSnapshots4 2023-07-12 15:40:27,103 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesWithSnapshots4 2023-07-12 15:40:27,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:27,103 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testTablesWithSnapshots4'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:27,103 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:27,203 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:27,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:27,204 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:27,204 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): b42cbcb78cb9d76bf763969942764e12/f1 is initiating major compaction (all files) 2023-07-12 15:40:27,205 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:27,205 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:27,205 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of b42cbcb78cb9d76bf763969942764e12/f1 in testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:27,205 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/83ebfcc786f64b08b873f7955371fc38, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/98f63fba38c1438b86bc9497933a78fa] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/.tmp, totalSize=238.8 K 2023-07-12 15:40:27,206 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 83ebfcc786f64b08b873f7955371fc38, keycount=410, bloomtype=ROW, size=119.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420022 2023-07-12 15:40:27,206 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 98f63fba38c1438b86bc9497933a78fa, keycount=410, bloomtype=ROW, size=119.5 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425285 2023-07-12 15:40:27,218 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): b42cbcb78cb9d76bf763969942764e12#f1#compaction#24 average throughput is 55.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:27,227 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/.tmp/f1/88f3c28c381b430e9f9eb8238e6e0549 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/88f3c28c381b430e9f9eb8238e6e0549 2023-07-12 15:40:27,234 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in b42cbcb78cb9d76bf763969942764e12/f1 of b42cbcb78cb9d76bf763969942764e12 into 88f3c28c381b430e9f9eb8238e6e0549(size=233.7 K), total size for store is 233.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:27,234 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for b42cbcb78cb9d76bf763969942764e12: 2023-07-12 15:40:27,234 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:27,234 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): 7e64a90200e73af2749582231be39d70/f1 is initiating major compaction (all files) 2023-07-12 15:40:27,234 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:27,234 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:27,235 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of 7e64a90200e73af2749582231be39d70/f1 in testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:27,235 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/a09b8ee0fae04691bcce0baeeec13876, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/d4eccc47ff9246bf9de59513ae282a9d] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/.tmp, totalSize=238.8 K 2023-07-12 15:40:27,236 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting a09b8ee0fae04691bcce0baeeec13876, keycount=410, bloomtype=ROW, size=119.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420019 2023-07-12 15:40:27,236 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting d4eccc47ff9246bf9de59513ae282a9d, keycount=410, bloomtype=ROW, size=119.5 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425284 2023-07-12 15:40:27,246 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): 7e64a90200e73af2749582231be39d70#f1#compaction#25 average throughput is 55.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:27,655 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/.tmp/f1/72ad8739cd3a44a6b1ed8a02085bf842 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/72ad8739cd3a44a6b1ed8a02085bf842 2023-07-12 15:40:27,660 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 7e64a90200e73af2749582231be39d70/f1 of 7e64a90200e73af2749582231be39d70 into 72ad8739cd3a44a6b1ed8a02085bf842(size=233.7 K), total size for store is 233.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:27,660 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for 7e64a90200e73af2749582231be39d70: 2023-07-12 15:40:27,661 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:27,661 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): aa169f7820d9c1d0085da2db0f07e85a/f1 is initiating major compaction (all files) 2023-07-12 15:40:27,661 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:27,661 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:27,662 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of aa169f7820d9c1d0085da2db0f07e85a/f1 in testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:27,662 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/67bebbbbc3ea4b81a180d0149480fd98, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/1065446dd9774726ba9ed61333fcbca3] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/.tmp, totalSize=238.3 K 2023-07-12 15:40:27,662 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 67bebbbbc3ea4b81a180d0149480fd98, keycount=409, bloomtype=ROW, size=119.0 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420027 2023-07-12 15:40:27,663 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 1065446dd9774726ba9ed61333fcbca3, keycount=409, bloomtype=ROW, size=119.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425287 2023-07-12 15:40:27,671 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): aa169f7820d9c1d0085da2db0f07e85a#f1#compaction#26 average throughput is 110.67 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:27,806 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x6c2a898d to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:27,809 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@679e21a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:27,810 DEBUG [hconnection-0x4f5a1ad5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:27,812 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:27,843 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/83ebfcc786f64b08b873f7955371fc38, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/98f63fba38c1438b86bc9497933a78fa] to archive 2023-07-12 15:40:27,844 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:27,846 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/83ebfcc786f64b08b873f7955371fc38 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/83ebfcc786f64b08b873f7955371fc38 2023-07-12 15:40:27,848 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/98f63fba38c1438b86bc9497933a78fa to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/f1/98f63fba38c1438b86bc9497933a78fa 2023-07-12 15:40:27,858 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/a09b8ee0fae04691bcce0baeeec13876, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/d4eccc47ff9246bf9de59513ae282a9d] to archive 2023-07-12 15:40:27,858 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:27,861 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/a09b8ee0fae04691bcce0baeeec13876 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/a09b8ee0fae04691bcce0baeeec13876 2023-07-12 15:40:27,862 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/d4eccc47ff9246bf9de59513ae282a9d to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/f1/d4eccc47ff9246bf9de59513ae282a9d 2023-07-12 15:40:28,081 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/.tmp/f1/6a33c623ae9d4b0b86b0da8dd9345e9a as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/6a33c623ae9d4b0b86b0da8dd9345e9a 2023-07-12 15:40:28,086 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in aa169f7820d9c1d0085da2db0f07e85a/f1 of aa169f7820d9c1d0085da2db0f07e85a into 6a33c623ae9d4b0b86b0da8dd9345e9a(size=233.2 K), total size for store is 233.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:28,086 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for aa169f7820d9c1d0085da2db0f07e85a: 2023-07-12 15:40:28,087 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:28,087 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): 964fa8dd605f4f75805eb52796edffe1/f1 is initiating major compaction (all files) 2023-07-12 15:40:28,087 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:28,087 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:28,088 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of 964fa8dd605f4f75805eb52796edffe1/f1 in testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:28,088 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/9ae9c618cb9046e1886b411877253f27, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/4233047df0154697ad99bf4f3c3311b4] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/.tmp, totalSize=238.8 K 2023-07-12 15:40:28,088 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 9ae9c618cb9046e1886b411877253f27, keycount=410, bloomtype=ROW, size=119.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420021 2023-07-12 15:40:28,089 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 4233047df0154697ad99bf4f3c3311b4, keycount=410, bloomtype=ROW, size=119.5 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425284 2023-07-12 15:40:28,097 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): 964fa8dd605f4f75805eb52796edffe1#f1#compaction#27 average throughput is 110.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:28,506 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/.tmp/f1/78ae854a40974093a2d42edfa7aaf1a9 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/78ae854a40974093a2d42edfa7aaf1a9 2023-07-12 15:40:28,511 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 964fa8dd605f4f75805eb52796edffe1/f1 of 964fa8dd605f4f75805eb52796edffe1 into 78ae854a40974093a2d42edfa7aaf1a9(size=233.7 K), total size for store is 233.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:28,511 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for 964fa8dd605f4f75805eb52796edffe1: 2023-07-12 15:40:28,512 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 0 store files, 0 compacting, 0 eligible, 16 blocking 2023-07-12 15:40:28,512 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:28,512 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): 6917e0806fe754e18b86baa9e5ba8962/f1 is initiating major compaction (all files) 2023-07-12 15:40:28,512 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:28,512 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:28,513 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of 6917e0806fe754e18b86baa9e5ba8962/f1 in testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:28,513 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/4eda41aa2caa4a7a891f05f35428d9e5, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/03af4ad782c14887b377de6535a570ae] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/.tmp, totalSize=238.8 K 2023-07-12 15:40:28,513 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 4eda41aa2caa4a7a891f05f35428d9e5, keycount=410, bloomtype=ROW, size=119.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420024 2023-07-12 15:40:28,514 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 03af4ad782c14887b377de6535a570ae, keycount=410, bloomtype=ROW, size=119.5 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425285 2023-07-12 15:40:28,521 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): 6917e0806fe754e18b86baa9e5ba8962#f1#compaction#28 average throughput is 110.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:28,867 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x03ffa615 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:28,872 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55d2cbcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:28,874 DEBUG [hconnection-0x4078ecbe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:28,875 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:28,929 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/.tmp/f1/456c4b9024b34946be1cc5fb944b77e8 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/456c4b9024b34946be1cc5fb944b77e8 2023-07-12 15:40:28,934 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 6917e0806fe754e18b86baa9e5ba8962/f1 of 6917e0806fe754e18b86baa9e5ba8962 into 456c4b9024b34946be1cc5fb944b77e8(size=233.7 K), total size for store is 233.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:28,934 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for 6917e0806fe754e18b86baa9e5ba8962: 2023-07-12 15:40:28,934 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:28,935 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): e925615655bfa901ee0af32750a71b5d/f1 is initiating major compaction (all files) 2023-07-12 15:40:28,935 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:28,935 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:28,935 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of e925615655bfa901ee0af32750a71b5d/f1 in testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:28,935 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/a19615574b964ff6bed94174a491efe1, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/5cfbaa27ae7b40f99490616ba3d3b7e0] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/.tmp, totalSize=238.8 K 2023-07-12 15:40:28,936 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting a19615574b964ff6bed94174a491efe1, keycount=410, bloomtype=ROW, size=119.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420025 2023-07-12 15:40:28,936 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 5cfbaa27ae7b40f99490616ba3d3b7e0, keycount=410, bloomtype=ROW, size=119.5 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425286 2023-07-12 15:40:28,946 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): e925615655bfa901ee0af32750a71b5d#f1#compaction#29 average throughput is 110.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:28,964 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesWithSnapshots4 moved into observance of table space quota. 2023-07-12 15:40:29,361 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/.tmp/f1/e265710942b8427aa1650ed0dc40f6c8 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/e265710942b8427aa1650ed0dc40f6c8 2023-07-12 15:40:29,367 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in e925615655bfa901ee0af32750a71b5d/f1 of e925615655bfa901ee0af32750a71b5d into e265710942b8427aa1650ed0dc40f6c8(size=233.7 K), total size for store is 233.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:29,367 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for e925615655bfa901ee0af32750a71b5d: 2023-07-12 15:40:29,367 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:29,368 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): 1aeb1aec90691c00f122ca5f9a55e824/f1 is initiating major compaction (all files) 2023-07-12 15:40:29,368 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:29,368 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:29,368 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of 1aeb1aec90691c00f122ca5f9a55e824/f1 in testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:29,368 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/af3faa3511264e3ea8671918e7831092, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/5f1c095d3dbb4b9aa5278baea8dc9376] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/.tmp, totalSize=238.3 K 2023-07-12 15:40:29,368 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting af3faa3511264e3ea8671918e7831092, keycount=409, bloomtype=ROW, size=119.0 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420025 2023-07-12 15:40:29,369 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 5f1c095d3dbb4b9aa5278baea8dc9376, keycount=409, bloomtype=ROW, size=119.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425286 2023-07-12 15:40:29,378 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): 1aeb1aec90691c00f122ca5f9a55e824#f1#compaction#30 average throughput is 110.67 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:29,389 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/.tmp/f1/c015d00ab55e47929b6a5952759b2451 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/c015d00ab55e47929b6a5952759b2451 2023-07-12 15:40:29,396 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 1aeb1aec90691c00f122ca5f9a55e824/f1 of 1aeb1aec90691c00f122ca5f9a55e824 into c015d00ab55e47929b6a5952759b2451(size=233.2 K), total size for store is 233.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:29,396 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for 1aeb1aec90691c00f122ca5f9a55e824: 2023-07-12 15:40:29,396 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:29,397 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): 6c78480cd12af27c2af33c2a96e94e5e/f1 is initiating major compaction (all files) 2023-07-12 15:40:29,397 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:29,397 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:29,398 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of 6c78480cd12af27c2af33c2a96e94e5e/f1 in testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:29,398 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/00d4d3d956e14a319b96f7059b87bf92, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/702a9d26e1a64fb4bd6f1367f4680066] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/.tmp, totalSize=467.2 K 2023-07-12 15:40:29,398 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 00d4d3d956e14a319b96f7059b87bf92, keycount=818, bloomtype=ROW, size=233.4 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420026 2023-07-12 15:40:29,399 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 702a9d26e1a64fb4bd6f1367f4680066, keycount=818, bloomtype=ROW, size=233.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425286 2023-07-12 15:40:29,411 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): 6c78480cd12af27c2af33c2a96e94e5e#f1#compaction#31 average throughput is 88.54 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:29,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2d693fd0 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:29,789 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@616dba2d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:29,790 DEBUG [hconnection-0x6bf3df6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:29,792 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:29,820 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/.tmp/f1/11a79c3ba187477bb0482ad54a334458 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/11a79c3ba187477bb0482ad54a334458 2023-07-12 15:40:29,826 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 6c78480cd12af27c2af33c2a96e94e5e/f1 of 6c78480cd12af27c2af33c2a96e94e5e into 11a79c3ba187477bb0482ad54a334458(size=461.6 K), total size for store is 461.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:29,826 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for 6c78480cd12af27c2af33c2a96e94e5e: 2023-07-12 15:40:29,826 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:29,826 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): 64ce4c9ba49998f908bf20dec4456a81/f1 is initiating major compaction (all files) 2023-07-12 15:40:29,826 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:29,826 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:29,826 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of 64ce4c9ba49998f908bf20dec4456a81/f1 in testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:29,827 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/7d3929ef689049d5854cbbfb57174568, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/9dd856192f854cc8b3ce36bc5960ae82] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/.tmp, totalSize=238.8 K 2023-07-12 15:40:29,827 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 7d3929ef689049d5854cbbfb57174568, keycount=410, bloomtype=ROW, size=119.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1689176420023 2023-07-12 15:40:29,827 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting 9dd856192f854cc8b3ce36bc5960ae82, keycount=410, bloomtype=ROW, size=119.5 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1689176425285 2023-07-12 15:40:29,838 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): 64ce4c9ba49998f908bf20dec4456a81#f1#compaction#32 average throughput is 110.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:29,854 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/.tmp/f1/c3dfae2fd5c34a18a3aaa594019bc8e3 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/c3dfae2fd5c34a18a3aaa594019bc8e3 2023-07-12 15:40:29,861 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in 64ce4c9ba49998f908bf20dec4456a81/f1 of 64ce4c9ba49998f908bf20dec4456a81 into c3dfae2fd5c34a18a3aaa594019bc8e3(size=233.7 K), total size for store is 233.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:29,861 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for 64ce4c9ba49998f908bf20dec4456a81: 2023-07-12 15:40:29,861 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(326): Checking for quota violation 2023-07-12 15:40:29,861 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:29,863 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$10(342): 2657236/3145728 SpaceQuotaStatus[policy=Optional.empty, inViolation=false] 2023-07-12 15:40:30,791 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1ffd54fa to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:30,794 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5db4afa3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:30,795 DEBUG [hconnection-0x30748fd7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:30,797 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:30,864 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$10(342): 2657236/3145728 SpaceQuotaStatus[policy=Optional.empty, inViolation=false] 2023-07-12 15:40:30,873 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesWithSnapshots4 moved into observance of table space quota. 2023-07-12 15:40:31,867 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$10(342): 2630694/3145728 SpaceQuotaStatus[policy=Optional.empty, inViolation=false] 2023-07-12 15:40:31,874 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x13c3da68 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:31,877 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c44b97e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:31,879 DEBUG [hconnection-0x18ebd864-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:31,880 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:32,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x2cefaf73 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:32,788 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1427db58, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:32,790 DEBUG [hconnection-0x20bb2886-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:32,792 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:32,868 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$10(342): 2630694/3145728 SpaceQuotaStatus[policy=Optional.empty, inViolation=false] 2023-07-12 15:40:32,930 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/67bebbbbc3ea4b81a180d0149480fd98, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/1065446dd9774726ba9ed61333fcbca3] to archive 2023-07-12 15:40:32,931 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:32,933 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/67bebbbbc3ea4b81a180d0149480fd98 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/67bebbbbc3ea4b81a180d0149480fd98 2023-07-12 15:40:32,934 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/1065446dd9774726ba9ed61333fcbca3 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/f1/1065446dd9774726ba9ed61333fcbca3 2023-07-12 15:40:32,943 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/9ae9c618cb9046e1886b411877253f27, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/4233047df0154697ad99bf4f3c3311b4] to archive 2023-07-12 15:40:32,944 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:32,945 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/9ae9c618cb9046e1886b411877253f27 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/9ae9c618cb9046e1886b411877253f27 2023-07-12 15:40:32,947 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/4233047df0154697ad99bf4f3c3311b4 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/f1/4233047df0154697ad99bf4f3c3311b4 2023-07-12 15:40:32,955 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/4eda41aa2caa4a7a891f05f35428d9e5, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/03af4ad782c14887b377de6535a570ae] to archive 2023-07-12 15:40:32,956 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:32,958 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/4eda41aa2caa4a7a891f05f35428d9e5 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/4eda41aa2caa4a7a891f05f35428d9e5 2023-07-12 15:40:32,959 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/03af4ad782c14887b377de6535a570ae to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/f1/03af4ad782c14887b377de6535a570ae 2023-07-12 15:40:32,971 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/a19615574b964ff6bed94174a491efe1, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/5cfbaa27ae7b40f99490616ba3d3b7e0] to archive 2023-07-12 15:40:32,979 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:32,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/a19615574b964ff6bed94174a491efe1 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/a19615574b964ff6bed94174a491efe1 2023-07-12 15:40:32,984 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/5cfbaa27ae7b40f99490616ba3d3b7e0 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/f1/5cfbaa27ae7b40f99490616ba3d3b7e0 2023-07-12 15:40:32,994 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/af3faa3511264e3ea8671918e7831092, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/5f1c095d3dbb4b9aa5278baea8dc9376] to archive 2023-07-12 15:40:32,995 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:32,997 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/af3faa3511264e3ea8671918e7831092 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/af3faa3511264e3ea8671918e7831092 2023-07-12 15:40:32,998 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/5f1c095d3dbb4b9aa5278baea8dc9376 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/f1/5f1c095d3dbb4b9aa5278baea8dc9376 2023-07-12 15:40:33,010 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/00d4d3d956e14a319b96f7059b87bf92, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/702a9d26e1a64fb4bd6f1367f4680066] to archive 2023-07-12 15:40:33,011 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:33,013 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/00d4d3d956e14a319b96f7059b87bf92 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/00d4d3d956e14a319b96f7059b87bf92 2023-07-12 15:40:33,015 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/702a9d26e1a64fb4bd6f1367f4680066 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/f1/702a9d26e1a64fb4bd6f1367f4680066 2023-07-12 15:40:33,027 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/7d3929ef689049d5854cbbfb57174568, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/9dd856192f854cc8b3ce36bc5960ae82] to archive 2023-07-12 15:40:33,028 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:33,030 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/7d3929ef689049d5854cbbfb57174568 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/7d3929ef689049d5854cbbfb57174568 2023-07-12 15:40:33,031 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/9dd856192f854cc8b3ce36bc5960ae82 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/f1/9dd856192f854cc8b3ce36bc5960ae82 2023-07-12 15:40:33,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7a7b2277 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:33,788 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a02acd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:33,790 DEBUG [hconnection-0x6e30d08e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:33,792 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:33,870 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$10(342): 2630694/3145728 SpaceQuotaStatus[policy=Optional.empty, inViolation=false] 2023-07-12 15:40:33,881 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(348): testTablesWithSnapshots4 moved into violation of table space quota with policy of NO_INSERTS 2023-07-12 15:40:34,789 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x628cd607 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:34,798 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a5fa38e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:34,800 DEBUG [hconnection-0x513b3d40-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:34,801 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:34,871 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$10(342): 3602162/3145728 SpaceQuotaStatus[policy=Optional[NO_INSERTS], inViolation=true] 2023-07-12 15:40:34,884 INFO [Listener at localhost/38241] hbase.ResourceChecker(175): after: quotas.TestSpaceQuotasWithSnapshots#testTablesWithSnapshots Thread=286 (was 318), OpenFileDescriptor=653 (was 699), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=115 (was 113) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=7441 (was 7018) - AvailableMemoryMB LEAK? - 2023-07-12 15:40:34,895 INFO [Listener at localhost/38241] hbase.ResourceChecker(147): before: quotas.TestSpaceQuotasWithSnapshots#testTablesInheritSnapshotSize Thread=284, OpenFileDescriptor=653, MaxFileDescriptor=60000, SystemLoadAverage=115, ProcessCount=166, AvailableMemoryMB=7440 2023-07-12 15:40:34,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.HMaster$4(2112): Client=jenkins//172.31.10.131 create 'testTablesInheritSnapshotSize5', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 15:40:34,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testTablesInheritSnapshotSize5 2023-07-12 15:40:34,899 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=49, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testTablesInheritSnapshotSize5 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 15:40:34,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(700): Client=jenkins//172.31.10.131 procedure request for creating table: namespace: "default" qualifier: "testTablesInheritSnapshotSize5" procId is: 49 2023-07-12 15:40:34,900 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=49, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testTablesInheritSnapshotSize5 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 15:40:34,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=49 2023-07-12 15:40:34,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547 2023-07-12 15:40:34,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(153): Directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547 empty. 2023-07-12 15:40:34,902 DEBUG [HFileArchiver-9] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547 2023-07-12 15:40:34,902 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTablesInheritSnapshotSize5 regions 2023-07-12 15:40:35,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=49 2023-07-12 15:40:35,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=49 2023-07-12 15:40:35,310 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp/data/default/testTablesInheritSnapshotSize5/.tabledesc/.tableinfo.0000000001 2023-07-12 15:40:35,311 INFO [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(7675): creating {ENCODED => a00963404870ea24c249dab457e70547, NAME => 'testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testTablesInheritSnapshotSize5', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.tmp 2023-07-12 15:40:35,318 DEBUG [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(866): Instantiated testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:35,318 DEBUG [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(1604): Closing a00963404870ea24c249dab457e70547, disabling compactions & flushes 2023-07-12 15:40:35,318 INFO [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(1626): Closing region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:35,318 DEBUG [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:35,318 DEBUG [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(1714): Acquired close lock on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. after waiting 0 ms 2023-07-12 15:40:35,318 DEBUG [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(1724): Updates disabled for region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:35,318 INFO [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(1838): Closed testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:35,318 DEBUG [RegionOpenAndInit-testTablesInheritSnapshotSize5-pool-0] regionserver.HRegion(1558): Region close journal for a00963404870ea24c249dab457e70547: 2023-07-12 15:40:35,320 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=49, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testTablesInheritSnapshotSize5 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 15:40:35,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689176435321"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689176435321"}]},"ts":"1689176435321"} 2023-07-12 15:40:35,323 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 15:40:35,323 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=49, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testTablesInheritSnapshotSize5 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 15:40:35,323 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTablesInheritSnapshotSize5","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176435323"}]},"ts":"1689176435323"} 2023-07-12 15:40:35,324 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTablesInheritSnapshotSize5, state=ENABLING in hbase:meta 2023-07-12 15:40:35,327 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesInheritSnapshotSize5, region=a00963404870ea24c249dab457e70547, ASSIGN}] 2023-07-12 15:40:35,329 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTablesInheritSnapshotSize5, region=a00963404870ea24c249dab457e70547, ASSIGN 2023-07-12 15:40:35,329 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTablesInheritSnapshotSize5, region=a00963404870ea24c249dab457e70547, ASSIGN; state=OFFLINE, location=jenkins-hbase5.apache.org,45089,1689176381369; forceNewPlan=false, retain=false 2023-07-12 15:40:35,480 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=a00963404870ea24c249dab457e70547, regionState=OPENING, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:35,481 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689176435480"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689176435480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689176435480"}]},"ts":"1689176435480"} 2023-07-12 15:40:35,482 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE; OpenRegionProcedure a00963404870ea24c249dab457e70547, server=jenkins-hbase5.apache.org,45089,1689176381369}] 2023-07-12 15:40:35,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=49 2023-07-12 15:40:35,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(130): Open testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:35,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a00963404870ea24c249dab457e70547, NAME => 'testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547.', STARTKEY => '', ENDKEY => ''} 2023-07-12 15:40:35,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTablesInheritSnapshotSize5 a00963404870ea24c249dab457e70547 2023-07-12 15:40:35,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(866): Instantiated testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 15:40:35,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7894): checking encryption for a00963404870ea24c249dab457e70547 2023-07-12 15:40:35,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(7897): checking classloading for a00963404870ea24c249dab457e70547 2023-07-12 15:40:35,637 INFO [StoreOpener-a00963404870ea24c249dab457e70547-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region a00963404870ea24c249dab457e70547 2023-07-12 15:40:35,638 DEBUG [StoreOpener-a00963404870ea24c249dab457e70547-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1 2023-07-12 15:40:35,638 DEBUG [StoreOpener-a00963404870ea24c249dab457e70547-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1 2023-07-12 15:40:35,639 INFO [StoreOpener-a00963404870ea24c249dab457e70547-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a00963404870ea24c249dab457e70547 columnFamilyName f1 2023-07-12 15:40:35,639 INFO [StoreOpener-a00963404870ea24c249dab457e70547-1] regionserver.HStore(310): Store=a00963404870ea24c249dab457e70547/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 15:40:35,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547 2023-07-12 15:40:35,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547 2023-07-12 15:40:35,642 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1055): writing seq id for a00963404870ea24c249dab457e70547 2023-07-12 15:40:35,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 15:40:35,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1072): Opened a00963404870ea24c249dab457e70547; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66835372, jitterRate=-0.004075348377227783}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 15:40:35,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(965): Region open journal for a00963404870ea24c249dab457e70547: 2023-07-12 15:40:35,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547., pid=51, masterSystemTime=1689176435633 2023-07-12 15:40:35,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:35,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase5:0-0] handler.AssignRegionHandler(158): Opened testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:35,647 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=a00963404870ea24c249dab457e70547, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:35,647 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1689176435647"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689176435647"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689176435647"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689176435647"}]},"ts":"1689176435647"} 2023-07-12 15:40:35,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-07-12 15:40:35,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; OpenRegionProcedure a00963404870ea24c249dab457e70547, server=jenkins-hbase5.apache.org,45089,1689176381369 in 166 msec 2023-07-12 15:40:35,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=50, resume processing ppid=49 2023-07-12 15:40:35,651 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testTablesInheritSnapshotSize5, region=a00963404870ea24c249dab457e70547, ASSIGN in 322 msec 2023-07-12 15:40:35,651 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=49, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testTablesInheritSnapshotSize5 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 15:40:35,651 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTablesInheritSnapshotSize5","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689176435651"}]},"ts":"1689176435651"} 2023-07-12 15:40:35,652 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testTablesInheritSnapshotSize5, state=ENABLED in hbase:meta 2023-07-12 15:40:35,654 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=49, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testTablesInheritSnapshotSize5 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 15:40:35,655 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; CreateTableProcedure table=testTablesInheritSnapshotSize5 in 756 msec 2023-07-12 15:40:35,795 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x36b43f34 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:35,799 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ee467b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:35,800 DEBUG [hconnection-0x715eaac7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:35,802 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:37396, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:36,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1230): Checking to see if procedure is done pid=49 2023-07-12 15:40:36,005 INFO [Listener at localhost/38241] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testTablesInheritSnapshotSize5, procId: 49 completed 2023-07-12 15:40:36,005 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(112): Writing data 2023-07-12 15:40:36,182 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 15:40:36,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:36,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testTablesInheritSnapshotSize5 2023-07-12 15:40:36,186 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testTablesInheritSnapshotSize5' 2023-07-12 15:40:36,186 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:36,187 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testTablesInheritSnapshotSize5' starting 'acquire' 2023-07-12 15:40:36,187 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testTablesInheritSnapshotSize5', kicking off acquire phase on members. 2023-07-12 15:40:36,187 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,187 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,188 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:36,188 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:36,189 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,189 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testTablesInheritSnapshotSize5 2023-07-12 15:40:36,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testTablesInheritSnapshotSize5 2023-07-12 15:40:36,190 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testTablesInheritSnapshotSize5' with timeout 60000ms 2023-07-12 15:40:36,190 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:36,190 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testTablesInheritSnapshotSize5' starting 'acquire' stage 2023-07-12 15:40:36,191 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:40:36,191 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:36,191 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:36,191 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. started... 2023-07-12 15:40:36,191 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a00963404870ea24c249dab457e70547 1/1 column families, dataSize=2.22 MB heapSize=2.81 MB 2023-07-12 15:40:36,611 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.22 MB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/.tmp/f1/f0347fa281dd41989aeb7ab925572fe8 2023-07-12 15:40:36,616 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/.tmp/f1/f0347fa281dd41989aeb7ab925572fe8 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/f0347fa281dd41989aeb7ab925572fe8 2023-07-12 15:40:36,620 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/f0347fa281dd41989aeb7ab925572fe8, entries=8192, sequenceid=165, filesize=2.3 M 2023-07-12 15:40:36,621 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.22 MB/2325418, heapSize ~2.81 MB/2949360, currentSize=0 B/0 for a00963404870ea24c249dab457e70547 in 430ms, sequenceid=165, compaction requested=false 2023-07-12 15:40:36,621 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTablesInheritSnapshotSize5' 2023-07-12 15:40:36,621 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a00963404870ea24c249dab457e70547: 2023-07-12 15:40:36,622 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:36,622 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:40:36,622 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:40:36,622 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,622 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testTablesInheritSnapshotSize5' locally acquired 2023-07-12 15:40:36,622 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testTablesInheritSnapshotSize5) in zk 2023-07-12 15:40:36,623 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,623 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:36,624 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:36,624 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testTablesInheritSnapshotSize5' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:36,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testTablesInheritSnapshotSize5' on coordinator 2023-07-12 15:40:36,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341fa98b[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:36,625 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testTablesInheritSnapshotSize5' starting 'in-barrier' execution. 2023-07-12 15:40:36,625 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,626 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,627 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testTablesInheritSnapshotSize5' received 'reached' from coordinator. 2023-07-12 15:40:36,627 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testTablesInheritSnapshotSize5' locally completed 2023-07-12 15:40:36,627 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testTablesInheritSnapshotSize5' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:36,627 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,627 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:36,628 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testTablesInheritSnapshotSize5' has notified controller of completion 2023-07-12 15:40:36,628 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,628 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,628 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testTablesInheritSnapshotSize5' completed. 2023-07-12 15:40:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testTablesInheritSnapshotSize5' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:36,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testTablesInheritSnapshotSize5', counting down latch. Waiting for 0 more 2023-07-12 15:40:36,631 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testTablesInheritSnapshotSize5' execution completed 2023-07-12 15:40:36,631 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:36,631 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:36,631 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testTablesInheritSnapshotSize5 2023-07-12 15:40:36,631 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testTablesInheritSnapshotSize5including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:36,634 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,634 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:36,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,634 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:36,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:36,634 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,635 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:36,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,640 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,640 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:36,640 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:36,640 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:36,640 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testTablesInheritSnapshotSize5' 2023-07-12 15:40:36,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:36,640 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:36,640 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:36,641 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,641 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testTablesInheritSnapshotSize5'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:36,641 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,641 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:36,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:36,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:36,641 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:36,742 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:36,743 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 15:40:36,743 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(121): Waiting until table size reflects written data 2023-07-12 15:40:36,744 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:36,745 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: null 2023-07-12 15:40:36,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x73781768 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:36,788 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695ce7f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:36,790 DEBUG [hconnection-0x3083a2a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:36,792 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:36,868 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesInheritSnapshotSize5 moved into observance of table space quota. 2023-07-12 15:40:37,246 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:40:37,747 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=0 B/1 GB] 2023-07-12 15:40:37,788 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1e097e3e to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:37,792 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ca1a02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:37,794 DEBUG [hconnection-0x30428545-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:37,795 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60478, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:37,881 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesInheritSnapshotSize5 moved into observance of table space quota. 2023-07-12 15:40:38,249 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:38,249 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [15,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:38,250 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:38,251 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=-1 2023-07-12 15:40:38,752 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:38,752 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:38,839 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3cc8ef72 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:38,842 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a97bc57, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:38,844 DEBUG [hconnection-0x1afa912d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:38,847 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60484, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:39,253 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:39,254 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:39,755 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:39,755 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:39,798 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x0244c69b to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:39,801 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ec8adff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:39,802 DEBUG [hconnection-0x3e0c40b9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:39,804 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:40,256 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:40,257 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:40,702 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36b43f34 to 127.0.0.1:51623 2023-07-12 15:40:40,702 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,702 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73781768 to 127.0.0.1:51623 2023-07-12 15:40:40,702 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,703 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e097e3e to 127.0.0.1:51623 2023-07-12 15:40:40,703 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,703 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cc8ef72 to 127.0.0.1:51623 2023-07-12 15:40:40,703 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,703 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0244c69b to 127.0.0.1:51623 2023-07-12 15:40:40,703 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,703 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cefaf73 to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,704 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a7b2277 to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,704 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x628cd607 to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,704 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d693fd0 to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,704 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ffd54fa to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,704 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13c3da68 to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,704 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c2a898d to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,704 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03ffa615 to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,704 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e301388 to 127.0.0.1:51623 2023-07-12 15:40:40,704 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,705 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x084dae70 to 127.0.0.1:51623 2023-07-12 15:40:40,705 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,705 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12b3e80c to 127.0.0.1:51623 2023-07-12 15:40:40,705 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,705 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72e3d45b to 127.0.0.1:51623 2023-07-12 15:40:40,705 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,705 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e3de1d0 to 127.0.0.1:51623 2023-07-12 15:40:40,705 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,705 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1972b744 to 127.0.0.1:51623 2023-07-12 15:40:40,705 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,705 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e0a95e8 to 127.0.0.1:51623 2023-07-12 15:40:40,705 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,706 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d0c3ece to 127.0.0.1:51623 2023-07-12 15:40:40,706 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:40,758 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.25 MB/1 GB] 2023-07-12 15:40:40,758 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$14(433): Last observed size=2359562 2023-07-12 15:40:40,759 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(135): Initial table size was 2359562 2023-07-12 15:40:40,759 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(137): Snapshot the table 2023-07-12 15:40:40,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1583): Client=jenkins//172.31.10.131 snapshot request for:{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 } 2023-07-12 15:40:40,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1689176440759 (current time:1689176440759). 2023-07-12 15:40:40,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-07-12 15:40:40,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] zookeeper.ReadOnlyZKClient(139): Connect 0x4544032b to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:40,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a9e3774, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:40,873 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x51f3aa7d to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:40,877 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b1d27b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:41,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:41,109 DEBUG [hconnection-0x5bfeed59-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:41,112 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:41,112 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:41,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4544032b to 127.0.0.1:51623 2023-07-12 15:40:41,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:41,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-07-12 15:40:41,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(648): Table enabled, starting distributed snapshots for { ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 } 2023-07-12 15:40:41,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure2.ProcedureExecutor(1029): Stored pid=52, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=EXCLUSIVE 2023-07-12 15:40:41,117 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=52, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=EXCLUSIVE 2023-07-12 15:40:41,117 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=52, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=EXCLUSIVE; timeout=600000, timestamp=1689177041117 2023-07-12 15:40:41,117 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(174): Running FLUSH table snapshot testTablesInheritSnapshotSize5_snapshot1 C_M_SNAPSHOT_TABLE on table testTablesInheritSnapshotSize5 2023-07-12 15:40:41,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(653): Started snapshot: { ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 } 2023-07-12 15:40:41,118 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 }'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:41,118 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#1) Sleeping: 100ms while waiting for snapshot completion. 2023-07-12 15:40:41,118 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=SHARED 2023-07-12 15:40:41,119 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=52, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=EXCLUSIVE 2023-07-12 15:40:41,119 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=EXCLUSIVE in 4 msec 2023-07-12 15:40:41,120 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=53, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=SHARED 2023-07-12 15:40:41,120 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=53, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=SHARED; timeout=600000, timestamp=1689177041120 2023-07-12 15:40:41,203 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 15:40:41,218 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:41,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 } is done 2023-07-12 15:40:41,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:41,220 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#2) Sleeping: 200ms while waiting for snapshot completion. 2023-07-12 15:40:41,420 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:41,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 } is done 2023-07-12 15:40:41,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:41,422 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#3) Sleeping: 300ms while waiting for snapshot completion. 2023-07-12 15:40:41,528 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] procedure.ProcedureCoordinator(165): Submitting procedure testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,528 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testTablesInheritSnapshotSize5_snapshot1' 2023-07-12 15:40:41,528 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 15:40:41,528 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testTablesInheritSnapshotSize5_snapshot1' starting 'acquire' 2023-07-12 15:40:41,528 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testTablesInheritSnapshotSize5_snapshot1', kicking off acquire phase on members. 2023-07-12 15:40:41,528 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,528 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,530 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 15:40:41,530 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 15:40:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:40:41,530 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,530 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 91 2023-07-12 15:40:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,531 DEBUG [zk-event-processor-pool-0] snapshot.RegionServerSnapshotManager(175): Launching subprocedure for snapshot testTablesInheritSnapshotSize5_snapshot1 from table testTablesInheritSnapshotSize5 type FLUSH 2023-07-12 15:40:41,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,531 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testTablesInheritSnapshotSize5_snapshot1' with timeout 300000ms 2023-07-12 15:40:41,531 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 300000 ms 2023-07-12 15:40:41,532 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testTablesInheritSnapshotSize5_snapshot1' starting 'acquire' stage 2023-07-12 15:40:41,532 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testTablesInheritSnapshotSize5_snapshot1' locally acquired 2023-07-12 15:40:41,532 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testTablesInheritSnapshotSize5_snapshot1) in zk 2023-07-12 15:40:41,533 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,533 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:41,534 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,534 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testTablesInheritSnapshotSize5_snapshot1' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:41,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testTablesInheritSnapshotSize5_snapshot1' on coordinator 2023-07-12 15:40:41,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1deeaabf[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:41,535 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testTablesInheritSnapshotSize5_snapshot1' starting 'in-barrier' execution. 2023-07-12 15:40:41,535 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,536 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,536 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testTablesInheritSnapshotSize5_snapshot1' received 'reached' from coordinator. 2023-07-12 15:40:41,536 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,536 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:41,536 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.FlushSnapshotSubprocedure(170): Flush Snapshot Tasks submitted for 1 regions 2023-07-12 15:40:41,536 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(301): Waiting for local region snapshots to finish. 2023-07-12 15:40:41,536 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Starting snapshot operation on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:41,537 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(110): Flush Snapshotting region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. started... 2023-07-12 15:40:41,537 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] regionserver.HRegion(2446): Flush status journal for a00963404870ea24c249dab457e70547: 2023-07-12 15:40:41,537 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(238): Storing 'testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547.' region-info for snapshot=testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,537 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(243): Creating references for hfiles 2023-07-12 15:40:41,537 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(253): Adding snapshot references for [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/f0347fa281dd41989aeb7ab925572fe8] hfiles 2023-07-12 15:40:41,537 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.SnapshotManifest(262): Adding reference for file (1/1): hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/f0347fa281dd41989aeb7ab925572fe8 for snapshot=testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,722 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:41,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 } is done 2023-07-12 15:40:41,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:41,723 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#4) Sleeping: 500ms while waiting for snapshot completion. 2023-07-12 15:40:41,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4c191f88 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:41,793 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a51f9d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:41,795 DEBUG [hconnection-0xaba7b7d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:41,797 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60526, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:41,942 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(137): ... Flush Snapshotting region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. completed. 2023-07-12 15:40:41,942 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-snapshot-pool-0] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(140): Closing snapshot operation on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:41,942 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(312): Completed 1/1 local region snapshots. 2023-07-12 15:40:41,942 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(314): Completed 1 local region snapshots. 2023-07-12 15:40:41,942 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(345): cancelling 0 tasks for snapshot jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,942 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testTablesInheritSnapshotSize5_snapshot1' locally completed 2023-07-12 15:40:41,942 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testTablesInheritSnapshotSize5_snapshot1' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:41,944 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testTablesInheritSnapshotSize5_snapshot1' has notified controller of completion 2023-07-12 15:40:41,944 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,944 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:41,944 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testTablesInheritSnapshotSize5_snapshot1' completed. 2023-07-12 15:40:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:41,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testTablesInheritSnapshotSize5_snapshot1' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:41,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testTablesInheritSnapshotSize5_snapshot1', counting down latch. Waiting for 0 more 2023-07-12 15:40:41,947 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testTablesInheritSnapshotSize5_snapshot1' execution completed 2023-07-12 15:40:41,947 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:41,947 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:41,947 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,947 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testTablesInheritSnapshotSize5_snapshot1including nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2023-07-12 15:40:41,949 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,949 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,949 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 15:40:41,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/online-snapshot 2023-07-12 15:40:41,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 15:40:41,950 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:40:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,951 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,953 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,953 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired 2023-07-12 15:40:41,953 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,953 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/abort 2023-07-12 15:40:41,953 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,953 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.EnabledTableSnapshotHandler(97): Done waiting - online snapshot for testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 because node does not exist (not an error) 2023-07-12 15:40:41,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/online-snapshot/acquired 2023-07-12 15:40:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 15:40:41,953 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,953 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:41,953 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:41,953 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,954 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,954 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:41,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/online-snapshot/abort 2023-07-12 15:40:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 15:40:41,955 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:42,223 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:42,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 } is done 2023-07-12 15:40:42,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(404): Snapshoting '{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 }' is still in progress! 2023-07-12 15:40:42,224 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2428): (#5) Sleeping: 1000ms while waiting for snapshot completion. 2023-07-12 15:40:42,368 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testTablesInheritSnapshotSize5_snapshot1 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:42,797 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x608d75fb to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:42,800 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1542fc79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:42,801 DEBUG [hconnection-0x2c7ca1c4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:42,802 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:43,184 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(229): Snapshot testTablesInheritSnapshotSize5_snapshot1 of table testTablesInheritSnapshotSize5 completed 2023-07-12 15:40:43,185 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:43,185 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/.hbase-snapshot/.tmp/testTablesInheritSnapshotSize5_snapshot1 2023-07-12 15:40:43,185 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase5:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running FLUSH table snapshot testTablesInheritSnapshotSize5_snapshot1 C_M_SNAPSHOT_TABLE on table testTablesInheritSnapshotSize5 at 1689176441117Consolidate snapshot: testTablesInheritSnapshotSize5_snapshot1 at 1689176441953 (+836 ms)Loading Region manifests for testTablesInheritSnapshotSize5_snapshot1 at 1689176441954 (+1 ms)Writing data manifest for testTablesInheritSnapshotSize5_snapshot1 at 1689176441957 (+3 ms)Verifying snapshot: testTablesInheritSnapshotSize5_snapshot1 at 1689176442363 (+406 ms)Snapshot testTablesInheritSnapshotSize5_snapshot1 of table testTablesInheritSnapshotSize5 completed at 1689176443184 (+821 ms) 2023-07-12 15:40:43,186 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=53, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=SHARED 2023-07-12 15:40:43,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testTablesInheritSnapshotSize5, type=SHARED in 2.0680 sec 2023-07-12 15:40:43,225 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-07-12 15:40:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1212): Checking to see if snapshot from request:{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 } is done 2023-07-12 15:40:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] snapshot.SnapshotManager(401): Snapshot '{ ss=testTablesInheritSnapshotSize5_snapshot1 table=testTablesInheritSnapshotSize5 type=FLUSH ttl=0 }' has completed, notifying client. 2023-07-12 15:40:43,226 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(143): Write more data 2023-07-12 15:40:43,412 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(539): Data was written to HBase 2023-07-12 15:40:43,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:43,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure testTablesInheritSnapshotSize5 was in running list but was completed. Accepting new attempt. 2023-07-12 15:40:43,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testTablesInheritSnapshotSize5 2023-07-12 15:40:43,415 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testTablesInheritSnapshotSize5' 2023-07-12 15:40:43,415 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:43,416 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testTablesInheritSnapshotSize5' starting 'acquire' 2023-07-12 15:40:43,416 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testTablesInheritSnapshotSize5', kicking off acquire phase on members. 2023-07-12 15:40:43,416 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,416 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,417 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:43,417 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:43,418 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,418 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testTablesInheritSnapshotSize5 2023-07-12 15:40:43,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testTablesInheritSnapshotSize5 2023-07-12 15:40:43,419 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testTablesInheritSnapshotSize5' with timeout 60000ms 2023-07-12 15:40:43,419 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:43,420 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testTablesInheritSnapshotSize5' starting 'acquire' stage 2023-07-12 15:40:43,420 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:40:43,420 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:43,420 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:43,420 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. started... 2023-07-12 15:40:43,420 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a00963404870ea24c249dab457e70547 1/1 column families, dataSize=2.22 MB heapSize=2.81 MB 2023-07-12 15:40:43,797 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x25a4291c to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:43,800 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c0d18c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:43,801 DEBUG [hconnection-0x2cc111ce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:43,803 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60556, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:43,842 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.22 MB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/.tmp/f1/ccbaf1660cb741f3a42fe8a1b7f35606 2023-07-12 15:40:43,847 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/.tmp/f1/ccbaf1660cb741f3a42fe8a1b7f35606 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/ccbaf1660cb741f3a42fe8a1b7f35606 2023-07-12 15:40:43,851 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/ccbaf1660cb741f3a42fe8a1b7f35606, entries=8192, sequenceid=330, filesize=2.3 M 2023-07-12 15:40:43,852 INFO [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.22 MB/2325418, heapSize ~2.81 MB/2949360, currentSize=0 B/0 for a00963404870ea24c249dab457e70547 in 431ms, sequenceid=330, compaction requested=false 2023-07-12 15:40:43,852 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a00963404870ea24c249dab457e70547: 2023-07-12 15:40:43,852 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:43,852 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:40:43,852 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:40:43,852 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,852 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testTablesInheritSnapshotSize5' locally acquired 2023-07-12 15:40:43,852 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testTablesInheritSnapshotSize5) in zk 2023-07-12 15:40:43,854 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,854 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:43,855 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,855 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testTablesInheritSnapshotSize5' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:43,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testTablesInheritSnapshotSize5' on coordinator 2023-07-12 15:40:43,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45567b01[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:43,856 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testTablesInheritSnapshotSize5' starting 'in-barrier' execution. 2023-07-12 15:40:43,857 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,859 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,859 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testTablesInheritSnapshotSize5' received 'reached' from coordinator. 2023-07-12 15:40:43,859 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,860 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:43,859 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testTablesInheritSnapshotSize5' locally completed 2023-07-12 15:40:43,860 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testTablesInheritSnapshotSize5' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:43,861 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,861 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testTablesInheritSnapshotSize5' has notified controller of completion 2023-07-12 15:40:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:43,861 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:43,862 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testTablesInheritSnapshotSize5' completed. 2023-07-12 15:40:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testTablesInheritSnapshotSize5' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:43,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testTablesInheritSnapshotSize5', counting down latch. Waiting for 0 more 2023-07-12 15:40:43,865 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testTablesInheritSnapshotSize5' execution completed 2023-07-12 15:40:43,865 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:43,865 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:43,865 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testTablesInheritSnapshotSize5 2023-07-12 15:40:43,865 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testTablesInheritSnapshotSize5including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:43,867 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,867 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:43,867 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:43,868 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,870 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,878 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,878 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:43,878 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:43,878 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:43,878 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:43,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testTablesInheritSnapshotSize5' 2023-07-12 15:40:43,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:43,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 because node does not exist (not an error) 2023-07-12 15:40:43,878 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,879 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,879 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,879 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,879 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testTablesInheritSnapshotSize5'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:43,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:43,879 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:43,879 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:43,979 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:43,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:43,981 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests(542): Data flushed to disk 2023-07-12 15:40:43,981 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(145): Flush the table 2023-07-12 15:40:43,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(936): Client=jenkins//172.31.10.131 procedure request for: flush-table-proc 2023-07-12 15:40:43,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(143): Procedure testTablesInheritSnapshotSize5 was in running list but was completed. Accepting new attempt. 2023-07-12 15:40:43,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] procedure.ProcedureCoordinator(165): Submitting procedure testTablesInheritSnapshotSize5 2023-07-12 15:40:43,984 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testTablesInheritSnapshotSize5' 2023-07-12 15:40:43,984 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:43,985 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testTablesInheritSnapshotSize5' starting 'acquire' 2023-07-12 15:40:43,985 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testTablesInheritSnapshotSize5', kicking off acquire phase on members. 2023-07-12 15:40:43,985 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,985 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,987 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,987 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:43,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:43,987 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,987 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-07-12 15:40:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-07-12 15:40:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testTablesInheritSnapshotSize5 2023-07-12 15:40:43,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testTablesInheritSnapshotSize5 is still present, removing 2023-07-12 15:40:43,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testTablesInheritSnapshotSize5 2023-07-12 15:40:43,988 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testTablesInheritSnapshotSize5' with timeout 60000ms 2023-07-12 15:40:43,988 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-07-12 15:40:43,989 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testTablesInheritSnapshotSize5' starting 'acquire' stage 2023-07-12 15:40:43,989 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-07-12 15:40:43,989 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-07-12 15:40:43,989 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:43,989 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. started... 2023-07-12 15:40:43,989 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a00963404870ea24c249dab457e70547: 2023-07-12 15:40:43,989 DEBUG [rs(jenkins-hbase5.apache.org,45089,1689176381369)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:43,989 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-07-12 15:40:43,989 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-07-12 15:40:43,990 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,990 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testTablesInheritSnapshotSize5' locally acquired 2023-07-12 15:40:43,990 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure (testTablesInheritSnapshotSize5) in zk 2023-07-12 15:40:43,991 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,991 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:43,991 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:43,991 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testTablesInheritSnapshotSize5' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-07-12 15:40:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:43,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase5.apache.org,45089,1689176381369' joining acquired barrier for procedure 'testTablesInheritSnapshotSize5' on coordinator 2023-07-12 15:40:43,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b127444[Count = 0] remaining members to acquire global barrier 2023-07-12 15:40:43,993 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testTablesInheritSnapshotSize5' starting 'in-barrier' execution. 2023-07-12 15:40:43,993 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,995 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:43,995 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testTablesInheritSnapshotSize5' received 'reached' from coordinator. 2023-07-12 15:40:43,995 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,995 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-07-12 15:40:43,995 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testTablesInheritSnapshotSize5' locally completed 2023-07-12 15:40:43,996 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testTablesInheritSnapshotSize5' completed for member 'jenkins-hbase5.apache.org,45089,1689176381369' in zk 2023-07-12 15:40:43,997 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,997 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testTablesInheritSnapshotSize5' has notified controller of completion 2023-07-12 15:40:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:43,997 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:43,997 DEBUG [member: 'jenkins-hbase5.apache.org,45089,1689176381369' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testTablesInheritSnapshotSize5' completed. 2023-07-12 15:40:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testTablesInheritSnapshotSize5' member 'jenkins-hbase5.apache.org,45089,1689176381369': 2023-07-12 15:40:44,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase5.apache.org,45089,1689176381369' released barrier for procedure'testTablesInheritSnapshotSize5', counting down latch. Waiting for 0 more 2023-07-12 15:40:44,000 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testTablesInheritSnapshotSize5' execution completed 2023-07-12 15:40:44,000 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-07-12 15:40:44,000 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-07-12 15:40:44,000 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testTablesInheritSnapshotSize5 2023-07-12 15:40:44,000 INFO [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testTablesInheritSnapshotSize5including nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-07-12 15:40:44,002 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,002 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,002 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-07-12 15:40:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-07-12 15:40:44,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-07-12 15:40:44,003 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-07-12 15:40:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:44,004 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-07-12 15:40:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testTablesInheritSnapshotSize5 2023-07-12 15:40:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:44,006 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:44,006 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,006 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-07-12 15:40:44,006 DEBUG [(jenkins-hbase5.apache.org,37087,1689176380200)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-07-12 15:40:44,006 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,006 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-07-12 15:40:44,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-07-12 15:40:44,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testTablesInheritSnapshotSize5' 2023-07-12 15:40:44,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 because node does not exist (not an error) 2023-07-12 15:40:44,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-07-12 15:40:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 15:40:44,006 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:44,007 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,007 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,007 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testTablesInheritSnapshotSize5 2023-07-12 15:40:44,007 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testTablesInheritSnapshotSize5'' to complete. (max 20000 ms per retry) 2023-07-12 15:40:44,007 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-07-12 15:40:44,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-07-12 15:40:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 15:40:44,107 DEBUG [Listener at localhost/38241] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-07-12 15:40:44,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37087] master.MasterRpcServices(1189): Checking to see if procedure from request:flush-table-proc is done 2023-07-12 15:40:44,109 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(147): Synchronously compacting the table 2023-07-12 15:40:44,109 DEBUG [Listener at localhost/38241] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-07-12 15:40:44,109 DEBUG [Listener at localhost/38241] regionserver.HStore(1912): a00963404870ea24c249dab457e70547/f1 is initiating major compaction (all files) 2023-07-12 15:40:44,109 INFO [Listener at localhost/38241] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 15:40:44,109 INFO [Listener at localhost/38241] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 15:40:44,109 INFO [Listener at localhost/38241] regionserver.HRegion(2259): Starting compaction of a00963404870ea24c249dab457e70547/f1 in testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:44,109 INFO [Listener at localhost/38241] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/f0347fa281dd41989aeb7ab925572fe8, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/ccbaf1660cb741f3a42fe8a1b7f35606] into tmpdir=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/.tmp, totalSize=4.5 M 2023-07-12 15:40:44,110 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting f0347fa281dd41989aeb7ab925572fe8, keycount=8192, bloomtype=ROW, size=2.3 M, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1689176436010 2023-07-12 15:40:44,110 DEBUG [Listener at localhost/38241] compactions.Compactor(207): Compacting ccbaf1660cb741f3a42fe8a1b7f35606, keycount=8192, bloomtype=ROW, size=2.3 M, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1689176443227 2023-07-12 15:40:44,149 INFO [Listener at localhost/38241] throttle.PressureAwareThroughputController(145): a00963404870ea24c249dab457e70547#f1#compaction#35 average throughput is 71.54 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-07-12 15:40:44,557 DEBUG [Listener at localhost/38241] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/.tmp/f1/53f51f92c881435489ec854e3db05893 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/53f51f92c881435489ec854e3db05893 2023-07-12 15:40:44,562 INFO [Listener at localhost/38241] regionserver.HStore(1652): Completed major compaction of 2 (all) file(s) in a00963404870ea24c249dab457e70547/f1 of a00963404870ea24c249dab457e70547 into 53f51f92c881435489ec854e3db05893(size=2.3 M), total size for store is 2.3 M. This selection was in queue for 0sec, and took 0sec to execute. 2023-07-12 15:40:44,563 DEBUG [Listener at localhost/38241] regionserver.HRegion(2289): Compaction status journal for a00963404870ea24c249dab457e70547: 2023-07-12 15:40:44,563 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(154): Waiting for the region reports to reflect the correct size, between (1585152, 2609152) 2023-07-12 15:40:44,563 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:44,827 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesInheritSnapshotSize5 moved into observance of table space quota. 2023-07-12 15:40:44,829 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x3b1ca7db to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:44,833 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79b01422, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:44,834 DEBUG [hconnection-0x50db04e1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:44,836 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60562, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:45,066 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [15,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:45,066 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=-1 2023-07-12 15:40:45,567 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369718 2023-07-12 15:40:45,824 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x0a1cfa80 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:45,827 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@620c1aea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:45,828 DEBUG [hconnection-0x3021a6f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:45,829 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:60574, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:45,926 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesInheritSnapshotSize5 moved into observance of table space quota. 2023-07-12 15:40:46,068 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369718 2023-07-12 15:40:46,569 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369718 2023-07-12 15:40:46,798 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7466c860 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:46,808 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4848741, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:46,811 DEBUG [hconnection-0x2492d0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:46,813 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43884, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:47,070 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369718 2023-07-12 15:40:47,572 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$15(466): Last observed size=2369718 2023-07-12 15:40:47,574 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(168): Last seen size: 2369718 2023-07-12 15:40:47,575 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [20,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:47,576 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.26 MB/1 GB] 2023-07-12 15:40:47,576 INFO [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots(183): Expecting table usage to be 2359562 + 2369718 = 4729280 2023-07-12 15:40:47,576 INFO [Listener at localhost/38241] hbase.Waiter(180): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 15:40:47,577 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.26 MB/1 GB] 2023-07-12 15:40:47,577 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$5(189): Checking for 4729280 == 2369718 2023-07-12 15:40:47,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x7fddbb15 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:47,789 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d334ce1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:47,791 DEBUG [hconnection-0x1d544-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:47,793 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:47,902 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/f0347fa281dd41989aeb7ab925572fe8, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/ccbaf1660cb741f3a42fe8a1b7f35606] to archive 2023-07-12 15:40:47,903 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-07-12 15:40:47,905 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/f0347fa281dd41989aeb7ab925572fe8 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/f0347fa281dd41989aeb7ab925572fe8 2023-07-12 15:40:47,906 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase5:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/ccbaf1660cb741f3a42fe8a1b7f35606 to hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/archive/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/f1/ccbaf1660cb741f3a42fe8a1b7f35606 2023-07-12 15:40:48,578 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=2.26 MB/1 GB] 2023-07-12 15:40:48,578 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$5(189): Checking for 4729280 == 2369718 2023-07-12 15:40:48,789 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x4f74a1c5 to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:48,799 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c87a717, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:48,800 DEBUG [hconnection-0x68a25519-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:48,801 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:48,897 DEBUG [master/jenkins-hbase5:0.Chore.1] quotas.QuotaObserverChore(336): testTablesInheritSnapshotSize5 moved into observance of table space quota. 2023-07-12 15:40:49,580 DEBUG [Listener at localhost/38241] quotas.SpaceQuotaHelperForTests$SpaceQuotaSnapshotPredicate(731): Saw quota snapshot for testTablesInheritSnapshotSize5: SpaceQuotaSnapshot[policy=SpaceQuotaStatus[policy=Optional.empty, inViolation=false], use=4.51 MB/1 GB] 2023-07-12 15:40:49,580 DEBUG [Listener at localhost/38241] quotas.TestSpaceQuotasWithSnapshots$5(189): Checking for 4729280 == 4729280 2023-07-12 15:40:49,592 INFO [Listener at localhost/38241] hbase.ResourceChecker(175): after: quotas.TestSpaceQuotasWithSnapshots#testTablesInheritSnapshotSize Thread=243 (was 284), OpenFileDescriptor=545 (was 653), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=97 (was 115), ProcessCount=164 (was 166), AvailableMemoryMB=7243 (was 7440) 2023-07-12 15:40:49,592 INFO [Listener at localhost/38241] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 15:40:49,593 INFO [Listener at localhost/38241] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 15:40:49,593 DEBUG [Listener at localhost/38241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0971df84 to 127.0.0.1:51623 2023-07-12 15:40:49,593 DEBUG [Listener at localhost/38241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:49,593 DEBUG [Listener at localhost/38241] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 15:40:49,593 DEBUG [Listener at localhost/38241] util.JVMClusterUtil(257): Found active master hash=207237979, stopped=false 2023-07-12 15:40:49,593 DEBUG [Listener at localhost/38241] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.quotas.MasterQuotasObserver 2023-07-12 15:40:49,594 INFO [Listener at localhost/38241] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:40:49,595 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:40:49,595 INFO [Listener at localhost/38241] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 15:40:49,595 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:40:49,595 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 15:40:49,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:40:49,596 DEBUG [Listener at localhost/38241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e245caf to 127.0.0.1:51623 2023-07-12 15:40:49,596 DEBUG [Listener at localhost/38241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:49,596 INFO [Listener at localhost/38241] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,45089,1689176381369' ***** 2023-07-12 15:40:49,596 INFO [Listener at localhost/38241] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 15:40:49,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 15:40:49,596 INFO [RS:0;jenkins-hbase5:45089] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 15:40:49,597 INFO [RS:0;jenkins-hbase5:45089] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 15:40:49,597 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 15:40:49,597 INFO [RS:0;jenkins-hbase5:45089] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 15:40:49,597 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 3d092a61087068ae644d6d865e43b012 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for a8c1301b634cdb9d0c9d3c8a94ed47b8 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for b5c1ee8ec2de1c08a223d44b92d4447c 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 98647cced3a0461ae7cfd8e4946afc6b 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for b42cbcb78cb9d76bf763969942764e12 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 7e64a90200e73af2749582231be39d70 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for aa169f7820d9c1d0085da2db0f07e85a 2023-07-12 15:40:49,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 3d092a61087068ae644d6d865e43b012, disabling compactions & flushes 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 964fa8dd605f4f75805eb52796edffe1 2023-07-12 15:40:49,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 16362efeb81ea80068441d436c3937dd 2023-07-12 15:40:49,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for a5559313088d907a5c06d3ba77e10f9b 2023-07-12 15:40:49,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. after waiting 0 ms 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 6917e0806fe754e18b86baa9e5ba8962 2023-07-12 15:40:49,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for a00963404870ea24c249dab457e70547 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 1aeb1aec90691c00f122ca5f9a55e824 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 6c78480cd12af27c2af33c2a96e94e5e 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 64ce4c9ba49998f908bf20dec4456a81 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:49,598 INFO [RS:0;jenkins-hbase5:45089] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 15:40:49,599 DEBUG [RS:0;jenkins-hbase5:45089] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3eb8bc3b to 127.0.0.1:51623 2023-07-12 15:40:49,599 DEBUG [RS:0;jenkins-hbase5:45089] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:49,599 INFO [RS:0;jenkins-hbase5:45089] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 15:40:49,599 INFO [RS:0;jenkins-hbase5:45089] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 15:40:49,599 INFO [RS:0;jenkins-hbase5:45089] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 15:40:49,599 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 15:40:49,599 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1474): Waiting on 17 regions to close 2023-07-12 15:40:49,600 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 15:40:49,600 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 15:40:49,600 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 15:40:49,600 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 15:40:49,600 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 15:40:49,600 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=22.20 KB heapSize=35.75 KB 2023-07-12 15:40:49,601 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1504): Waiting on 1588230740, 16362efeb81ea80068441d436c3937dd, 1aeb1aec90691c00f122ca5f9a55e824, 3d092a61087068ae644d6d865e43b012, 64ce4c9ba49998f908bf20dec4456a81, 6917e0806fe754e18b86baa9e5ba8962, 6c78480cd12af27c2af33c2a96e94e5e, 7e64a90200e73af2749582231be39d70, 964fa8dd605f4f75805eb52796edffe1, 98647cced3a0461ae7cfd8e4946afc6b, a00963404870ea24c249dab457e70547, a5559313088d907a5c06d3ba77e10f9b, a8c1301b634cdb9d0c9d3c8a94ed47b8, aa169f7820d9c1d0085da2db0f07e85a, b42cbcb78cb9d76bf763969942764e12, b5c1ee8ec2de1c08a223d44b92d4447c, e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:49,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/ns2/testNamespacesInheritSnapshotSize3/3d092a61087068ae644d6d865e43b012/recovered.edits/336.seqid, newMaxSeqId=336, maxSeqId=1 2023-07-12 15:40:49,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:49,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 3d092a61087068ae644d6d865e43b012: 2023-07-12 15:40:49,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed ns2:testNamespacesInheritSnapshotSize3,,1689176401527.3d092a61087068ae644d6d865e43b012. 2023-07-12 15:40:49,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing a8c1301b634cdb9d0c9d3c8a94ed47b8, disabling compactions & flushes 2023-07-12 15:40:49,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:40:49,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:40:49,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. after waiting 0 ms 2023-07-12 15:40:49,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:40:49,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing a8c1301b634cdb9d0c9d3c8a94ed47b8 1/1 column families, dataSize=111 B heapSize=600 B 2023-07-12 15:40:49,609 INFO [regionserver/jenkins-hbase5:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:40:49,612 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=21.37 KB at sequenceid=57 (bloomFilter=false), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/.tmp/info/7614e721cf0c4440b95c4015c533084d 2023-07-12 15:40:49,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=111 B at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8/.tmp/info/a129ffe54d59486e9549d52d0a2ced65 2023-07-12 15:40:49,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8/.tmp/info/a129ffe54d59486e9549d52d0a2ced65 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8/info/a129ffe54d59486e9549d52d0a2ced65 2023-07-12 15:40:49,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8/info/a129ffe54d59486e9549d52d0a2ced65, entries=3, sequenceid=7, filesize=4.8 K 2023-07-12 15:40:49,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~111 B/111, heapSize ~584 B/584, currentSize=0 B/0 for a8c1301b634cdb9d0c9d3c8a94ed47b8 in 17ms, sequenceid=7, compaction requested=false 2023-07-12 15:40:49,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/namespace/a8c1301b634cdb9d0c9d3c8a94ed47b8/recovered.edits/10.seqid, newMaxSeqId=10, maxSeqId=1 2023-07-12 15:40:49,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:40:49,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for a8c1301b634cdb9d0c9d3c8a94ed47b8: 2023-07-12 15:40:49,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689176383355.a8c1301b634cdb9d0c9d3c8a94ed47b8. 2023-07-12 15:40:49,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing b5c1ee8ec2de1c08a223d44b92d4447c, disabling compactions & flushes 2023-07-12 15:40:49,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:40:49,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:40:49,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. after waiting 0 ms 2023-07-12 15:40:49,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:40:49,646 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace0/b5c1ee8ec2de1c08a223d44b92d4447c/recovered.edits/334.seqid, newMaxSeqId=334, maxSeqId=1 2023-07-12 15:40:49,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:40:49,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for b5c1ee8ec2de1c08a223d44b92d4447c: 2023-07-12 15:40:49,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testRematerializedTablesDoNoInheritSpace0,,1689176385678.b5c1ee8ec2de1c08a223d44b92d4447c. 2023-07-12 15:40:49,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 98647cced3a0461ae7cfd8e4946afc6b, disabling compactions & flushes 2023-07-12 15:40:49,648 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:40:49,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:40:49,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. after waiting 0 ms 2023-07-12 15:40:49,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:40:49,651 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testRematerializedTablesDoNoInheritSpace1/98647cced3a0461ae7cfd8e4946afc6b/recovered.edits/169.seqid, newMaxSeqId=169, maxSeqId=165 2023-07-12 15:40:49,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:40:49,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 98647cced3a0461ae7cfd8e4946afc6b: 2023-07-12 15:40:49,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testRematerializedTablesDoNoInheritSpace1,,1689176385678.98647cced3a0461ae7cfd8e4946afc6b. 2023-07-12 15:40:49,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing b42cbcb78cb9d76bf763969942764e12, disabling compactions & flushes 2023-07-12 15:40:49,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:49,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:49,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. after waiting 0 ms 2023-07-12 15:40:49,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:49,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/b42cbcb78cb9d76bf763969942764e12/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:49,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:49,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for b42cbcb78cb9d76bf763969942764e12: 2023-07-12 15:40:49,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,2,1689176415889.b42cbcb78cb9d76bf763969942764e12. 2023-07-12 15:40:49,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 7e64a90200e73af2749582231be39d70, disabling compactions & flushes 2023-07-12 15:40:49,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:49,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:49,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. after waiting 0 ms 2023-07-12 15:40:49,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:49,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/7e64a90200e73af2749582231be39d70/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:49,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:49,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 7e64a90200e73af2749582231be39d70: 2023-07-12 15:40:49,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,0,1689176415889.7e64a90200e73af2749582231be39d70. 2023-07-12 15:40:49,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing aa169f7820d9c1d0085da2db0f07e85a, disabling compactions & flushes 2023-07-12 15:40:49,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:49,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:49,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. after waiting 0 ms 2023-07-12 15:40:49,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:49,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/aa169f7820d9c1d0085da2db0f07e85a/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:49,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:49,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for aa169f7820d9c1d0085da2db0f07e85a: 2023-07-12 15:40:49,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,9,1689176415889.aa169f7820d9c1d0085da2db0f07e85a. 2023-07-12 15:40:49,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 964fa8dd605f4f75805eb52796edffe1, disabling compactions & flushes 2023-07-12 15:40:49,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:49,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:49,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. after waiting 0 ms 2023-07-12 15:40:49,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:49,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/964fa8dd605f4f75805eb52796edffe1/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:49,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:49,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 964fa8dd605f4f75805eb52796edffe1: 2023-07-12 15:40:49,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,1,1689176415889.964fa8dd605f4f75805eb52796edffe1. 2023-07-12 15:40:49,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 16362efeb81ea80068441d436c3937dd, disabling compactions & flushes 2023-07-12 15:40:49,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:49,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:49,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. after waiting 0 ms 2023-07-12 15:40:49,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:49,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/16362efeb81ea80068441d436c3937dd/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 15:40:49,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:49,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 16362efeb81ea80068441d436c3937dd: 2023-07-12 15:40:49,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,,1689176415889.16362efeb81ea80068441d436c3937dd. 2023-07-12 15:40:49,671 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing a5559313088d907a5c06d3ba77e10f9b, disabling compactions & flushes 2023-07-12 15:40:49,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:40:49,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:40:49,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. after waiting 0 ms 2023-07-12 15:40:49,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:40:49,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2745): Flushing a5559313088d907a5c06d3ba77e10f9b 2/2 column families, dataSize=19.47 KB heapSize=39.68 KB 2023-07-12 15:40:49,785 DEBUG [master/jenkins-hbase5:0.Chore.1] zookeeper.ReadOnlyZKClient(139): Connect 0x1146426e to 127.0.0.1:51623 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 15:40:49,789 DEBUG [master/jenkins-hbase5:0.Chore.1] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65ee4dab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 15:40:49,790 DEBUG [hconnection-0x105b8125-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 15:40:49,792 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.10.131:43924, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 15:40:49,792 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45089] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase5.apache.org,45089,1689176381369 stopping at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1527) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:40:49,793 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45089] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 131 connection: 172.31.10.131:43924 deadline: 1689176509792, exception=org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase5.apache.org,45089,1689176381369 stopping 2023-07-12 15:40:49,801 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1504): Waiting on 1588230740, 1aeb1aec90691c00f122ca5f9a55e824, 64ce4c9ba49998f908bf20dec4456a81, 6917e0806fe754e18b86baa9e5ba8962, 6c78480cd12af27c2af33c2a96e94e5e, a00963404870ea24c249dab457e70547, a5559313088d907a5c06d3ba77e10f9b, e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:49,898 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45089] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase5.apache.org,45089,1689176381369 stopping at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1527) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:40:49,898 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45089] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 131 connection: 172.31.10.131:43924 deadline: 1689176509790, exception=org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase5.apache.org,45089,1689176381369 stopping 2023-07-12 15:40:49,927 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 15:40:49,927 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 15:40:49,927 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: SpaceQuotaRefresherChore was stopped 2023-07-12 15:40:49,927 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: FileSystemUtilizationChore was stopped 2023-07-12 15:40:49,927 INFO [regionserver/jenkins-hbase5:0.Chore.1] hbase.ScheduledChore(146): Chore: RegionSizeReportingChore was stopped 2023-07-12 15:40:50,002 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1504): Waiting on 1588230740, 1aeb1aec90691c00f122ca5f9a55e824, 64ce4c9ba49998f908bf20dec4456a81, 6917e0806fe754e18b86baa9e5ba8962, 6c78480cd12af27c2af33c2a96e94e5e, a00963404870ea24c249dab457e70547, a5559313088d907a5c06d3ba77e10f9b, e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:50,030 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=848 B at sequenceid=57 (bloomFilter=false), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/.tmp/table/385f6cd2159648bb848fb220ac4d5169 2023-07-12 15:40:50,035 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/.tmp/info/7614e721cf0c4440b95c4015c533084d as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/info/7614e721cf0c4440b95c4015c533084d 2023-07-12 15:40:50,039 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/info/7614e721cf0c4440b95c4015c533084d, entries=160, sequenceid=57, filesize=26.5 K 2023-07-12 15:40:50,040 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/.tmp/table/385f6cd2159648bb848fb220ac4d5169 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/table/385f6cd2159648bb848fb220ac4d5169 2023-07-12 15:40:50,043 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/table/385f6cd2159648bb848fb220ac4d5169, entries=14, sequenceid=57, filesize=5.4 K 2023-07-12 15:40:50,044 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~22.20 KB/22728, heapSize ~35.47 KB/36320, currentSize=0 B/0 for 1588230740 in 444ms, sequenceid=57, compaction requested=false 2023-07-12 15:40:50,049 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/meta/1588230740/recovered.edits/60.seqid, newMaxSeqId=60, maxSeqId=1 2023-07-12 15:40:50,050 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:40:50,050 INFO [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 15:40:50,050 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 15:40:50,050 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 15:40:50,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=328 B at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/.tmp/q/8d92395453ef4809bc6addc3e3cb84a7 2023-07-12 15:40:50,100 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45089] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase5.apache.org,45089,1689176381369 stopping at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1527) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:40:50,101 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=45089] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 131 connection: 172.31.10.131:43924 deadline: 1689176509790, exception=org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase5.apache.org,45089,1689176381369 stopping 2023-07-12 15:40:50,202 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1504): Waiting on 1aeb1aec90691c00f122ca5f9a55e824, 64ce4c9ba49998f908bf20dec4456a81, 6917e0806fe754e18b86baa9e5ba8962, 6c78480cd12af27c2af33c2a96e94e5e, a00963404870ea24c249dab457e70547, a5559313088d907a5c06d3ba77e10f9b, e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:50,365 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 15:40:50,366 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.quotas.MasterQuotasObserver 2023-07-12 15:40:50,402 DEBUG [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1504): Waiting on 1aeb1aec90691c00f122ca5f9a55e824, 64ce4c9ba49998f908bf20dec4456a81, 6917e0806fe754e18b86baa9e5ba8962, 6c78480cd12af27c2af33c2a96e94e5e, a00963404870ea24c249dab457e70547, a5559313088d907a5c06d3ba77e10f9b, e925615655bfa901ee0af32750a71b5d 2023-07-12 15:40:50,402 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45089] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase5.apache.org,45089,1689176381369 stopping at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1527) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 15:40:50,403 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=45089] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 131 connection: 172.31.10.131:43924 deadline: 1689176509790, exception=org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server jenkins-hbase5.apache.org,45089,1689176381369 stopping 2023-07-12 15:40:50,488 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=19.15 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/.tmp/u/132d049b02ba4024bd3ecd3962ef9833 2023-07-12 15:40:50,492 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/.tmp/q/8d92395453ef4809bc6addc3e3cb84a7 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/q/8d92395453ef4809bc6addc3e3cb84a7 2023-07-12 15:40:50,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/q/8d92395453ef4809bc6addc3e3cb84a7, entries=5, sequenceid=222, filesize=5.1 K 2023-07-12 15:40:50,496 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/.tmp/u/132d049b02ba4024bd3ecd3962ef9833 as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/u/132d049b02ba4024bd3ecd3962ef9833 2023-07-12 15:40:50,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/u/132d049b02ba4024bd3ecd3962ef9833, entries=11, sequenceid=222, filesize=5.7 K 2023-07-12 15:40:50,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~19.47 KB/19940, heapSize ~39.65 KB/40600, currentSize=0 B/0 for a5559313088d907a5c06d3ba77e10f9b in 828ms, sequenceid=222, compaction requested=false 2023-07-12 15:40:50,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/hbase/quota/a5559313088d907a5c06d3ba77e10f9b/recovered.edits/225.seqid, newMaxSeqId=225, maxSeqId=1 2023-07-12 15:40:50,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:40:50,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for a5559313088d907a5c06d3ba77e10f9b: 2023-07-12 15:40:50,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed hbase:quota,,1689176384772.a5559313088d907a5c06d3ba77e10f9b. 2023-07-12 15:40:50,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 6917e0806fe754e18b86baa9e5ba8962, disabling compactions & flushes 2023-07-12 15:40:50,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:50,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:50,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. after waiting 0 ms 2023-07-12 15:40:50,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:50,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6917e0806fe754e18b86baa9e5ba8962/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:50,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:50,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 6917e0806fe754e18b86baa9e5ba8962: 2023-07-12 15:40:50,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,4,1689176415889.6917e0806fe754e18b86baa9e5ba8962. 2023-07-12 15:40:50,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing e925615655bfa901ee0af32750a71b5d, disabling compactions & flushes 2023-07-12 15:40:50,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:50,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:50,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. after waiting 0 ms 2023-07-12 15:40:50,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:50,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/e925615655bfa901ee0af32750a71b5d/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:50,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:50,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for e925615655bfa901ee0af32750a71b5d: 2023-07-12 15:40:50,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,5,1689176415889.e925615655bfa901ee0af32750a71b5d. 2023-07-12 15:40:50,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing a00963404870ea24c249dab457e70547, disabling compactions & flushes 2023-07-12 15:40:50,514 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:50,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:50,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. after waiting 0 ms 2023-07-12 15:40:50,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:50,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesInheritSnapshotSize5/a00963404870ea24c249dab457e70547/recovered.edits/335.seqid, newMaxSeqId=335, maxSeqId=1 2023-07-12 15:40:50,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:50,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for a00963404870ea24c249dab457e70547: 2023-07-12 15:40:50,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesInheritSnapshotSize5,,1689176434897.a00963404870ea24c249dab457e70547. 2023-07-12 15:40:50,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 1aeb1aec90691c00f122ca5f9a55e824, disabling compactions & flushes 2023-07-12 15:40:50,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:50,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:50,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. after waiting 0 ms 2023-07-12 15:40:50,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:50,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/1aeb1aec90691c00f122ca5f9a55e824/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:50,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:50,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 1aeb1aec90691c00f122ca5f9a55e824: 2023-07-12 15:40:50,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,6,1689176415889.1aeb1aec90691c00f122ca5f9a55e824. 2023-07-12 15:40:50,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 6c78480cd12af27c2af33c2a96e94e5e, disabling compactions & flushes 2023-07-12 15:40:50,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:50,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:50,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. after waiting 0 ms 2023-07-12 15:40:50,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:50,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/6c78480cd12af27c2af33c2a96e94e5e/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:50,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:50,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 6c78480cd12af27c2af33c2a96e94e5e: 2023-07-12 15:40:50,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,7,1689176415889.6c78480cd12af27c2af33c2a96e94e5e. 2023-07-12 15:40:50,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1604): Closing 64ce4c9ba49998f908bf20dec4456a81, disabling compactions & flushes 2023-07-12 15:40:50,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1626): Closing region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:50,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:50,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1714): Acquired close lock on testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. after waiting 0 ms 2023-07-12 15:40:50,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1724): Updates disabled for region testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:50,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/data/default/testTablesWithSnapshots4/64ce4c9ba49998f908bf20dec4456a81/recovered.edits/175.seqid, newMaxSeqId=175, maxSeqId=1 2023-07-12 15:40:50,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1838): Closed testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:50,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] regionserver.HRegion(1558): Region close journal for 64ce4c9ba49998f908bf20dec4456a81: 2023-07-12 15:40:50,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase5:0-0] handler.CloseRegionHandler(117): Closed testTablesWithSnapshots4,3,1689176415889.64ce4c9ba49998f908bf20dec4456a81. 2023-07-12 15:40:50,602 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,45089,1689176381369; all regions closed. 2023-07-12 15:40:50,602 DEBUG [RS:0;jenkins-hbase5:45089] quotas.QuotaCache(100): Stopping QuotaRefresherChore chore. 2023-07-12 15:40:50,610 DEBUG [RS:0;jenkins-hbase5:45089] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/oldWALs 2023-07-12 15:40:50,610 INFO [RS:0;jenkins-hbase5:45089] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C45089%2C1689176381369.meta:.meta(num 1689176383107) 2023-07-12 15:40:50,613 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/WALs/jenkins-hbase5.apache.org,45089,1689176381369/jenkins-hbase5.apache.org%2C45089%2C1689176381369.1689176383009 not finished, retry = 0 2023-07-12 15:40:50,715 DEBUG [RS:0;jenkins-hbase5:45089] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/oldWALs 2023-07-12 15:40:50,715 INFO [RS:0;jenkins-hbase5:45089] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase5.apache.org%2C45089%2C1689176381369:(num 1689176383009) 2023-07-12 15:40:50,715 DEBUG [RS:0;jenkins-hbase5:45089] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,715 INFO [RS:0;jenkins-hbase5:45089] regionserver.LeaseManager(133): Closed leases 2023-07-12 15:40:50,716 INFO [RS:0;jenkins-hbase5:45089] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase5:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=5000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 15:40:50,716 INFO [regionserver/jenkins-hbase5:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:40:50,717 INFO [RS:0;jenkins-hbase5:45089] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:45089 2023-07-12 15:40:50,857 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f74a1c5 to 127.0.0.1:51623 2023-07-12 15:40:50,857 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,857 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7466c860 to 127.0.0.1:51623 2023-07-12 15:40:50,857 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,857 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fddbb15 to 127.0.0.1:51623 2023-07-12 15:40:50,857 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,858 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b1ca7db to 127.0.0.1:51623 2023-07-12 15:40:50,858 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,858 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a1cfa80 to 127.0.0.1:51623 2023-07-12 15:40:50,858 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,858 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x608d75fb to 127.0.0.1:51623 2023-07-12 15:40:50,858 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,858 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25a4291c to 127.0.0.1:51623 2023-07-12 15:40:50,858 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,858 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51f3aa7d to 127.0.0.1:51623 2023-07-12 15:40:50,858 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,858 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c191f88 to 127.0.0.1:51623 2023-07-12 15:40:50,858 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,860 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase5.apache.org,45089,1689176381369 2023-07-12 15:40:50,860 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:40:50,863 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 15:40:50,963 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase5.apache.org,45089,1689176381369] 2023-07-12 15:40:50,963 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase5.apache.org,45089,1689176381369; numProcessing=1 2023-07-12 15:40:50,964 WARN [RS-EventLoopGroup-3-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase5.apache.org/172.31.10.131:45089 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:45089 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:40:50,964 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase5.apache.org,45089,1689176381369 already deleted, retry=false 2023-07-12 15:40:50,964 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase5.apache.org,45089,1689176381369 expired; onlineServers=0 2023-07-12 15:40:50,965 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase5.apache.org,37087,1689176380200' ***** 2023-07-12 15:40:50,965 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 15:40:50,965 DEBUG [RS-EventLoopGroup-3-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase5.apache.org/172.31.10.131:45089 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase5.apache.org/172.31.10.131:45089 2023-07-12 15:40:50,965 DEBUG [M:0;jenkins-hbase5:37087] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d673960, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase5.apache.org/172.31.10.131:0 2023-07-12 15:40:50,965 INFO [M:0;jenkins-hbase5:37087] regionserver.HRegionServer(1144): stopping server jenkins-hbase5.apache.org,37087,1689176380200 2023-07-12 15:40:50,965 INFO [M:0;jenkins-hbase5:37087] regionserver.HRegionServer(1170): stopping server jenkins-hbase5.apache.org,37087,1689176380200; all regions closed. 2023-07-12 15:40:50,965 DEBUG [M:0;jenkins-hbase5:37087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 15:40:50,965 DEBUG [M:0;jenkins-hbase5:37087] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 15:40:50,966 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 15:40:50,966 DEBUG [M:0;jenkins-hbase5:37087] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 15:40:50,966 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689176382587] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.small.0-1689176382587,5,FailOnTimeoutGroup] 2023-07-12 15:40:50,966 DEBUG [master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689176382581] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase5:0:becomeActiveMaster-HFileCleaner.large.0-1689176382581,5,FailOnTimeoutGroup] 2023-07-12 15:40:50,966 INFO [M:0;jenkins-hbase5:37087] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 15:40:50,966 INFO [M:0;jenkins-hbase5:37087] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 15:40:50,966 WARN [master/jenkins-hbase5:0.Chore.1] quotas.QuotaRetriever$Iter(131): java.io.InterruptedIOException at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:253) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:266) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:309) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:594) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:989) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:866) at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:319) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:157) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:266) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:309) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:594) at org.apache.hadoop.hbase.quotas.QuotaRetriever.next(QuotaRetriever.java:103) at org.apache.hadoop.hbase.quotas.QuotaRetriever$Iter.(QuotaRetriever.java:129) at org.apache.hadoop.hbase.quotas.QuotaRetriever.iterator(QuotaRetriever.java:121) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsToComputeSize(SnapshotQuotaObserverChore.java:171) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:106) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:40:50,966 INFO [M:0;jenkins-hbase5:37087] hbase.ChoreService(369): Chore service for: master/jenkins-hbase5:0 had [ScheduledChore name=QuotaObserverChore, period=1000, unit=MILLISECONDS] on shutdown 2023-07-12 15:40:50,966 DEBUG [M:0;jenkins-hbase5:37087] master.HMaster(1512): Stopping service threads 2023-07-12 15:40:50,966 WARN [master/jenkins-hbase5:0.Chore.1] quotas.SnapshotQuotaObserverChore(98): Failed to compute the size of snapshots, will retry org.apache.hadoop.hbase.exceptions.MasterStoppedException at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2843) at org.apache.hadoop.hbase.master.MasterRpcServices.getCompletedSnapshots(MasterRpcServices.java:1002) at org.apache.hadoop.hbase.client.ShortCircuitMasterConnection.getCompletedSnapshots(ShortCircuitMasterConnection.java:496) at org.apache.hadoop.hbase.client.HBaseAdmin$70.rpcCall(HBaseAdmin.java:2793) at org.apache.hadoop.hbase.client.HBaseAdmin$70.rpcCall(HBaseAdmin.java:2789) at org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:99) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2954) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2946) at org.apache.hadoop.hbase.client.HBaseAdmin.listSnapshots(HBaseAdmin.java:2788) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsFromTables(SnapshotQuotaObserverChore.java:199) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsToComputeSize(SnapshotQuotaObserverChore.java:188) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:106) at org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:93) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-07-12 15:40:50,967 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 15:40:50,967 INFO [M:0;jenkins-hbase5:37087] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 15:40:50,967 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 15:40:50,967 ERROR [M:0;jenkins-hbase5:37087] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-9,5,PEWorkerGroup] 2023-07-12 15:40:50,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 15:40:50,968 INFO [M:0;jenkins-hbase5:37087] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 15:40:50,968 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 15:40:50,968 DEBUG [M:0;jenkins-hbase5:37087] zookeeper.ZKUtil(398): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 15:40:50,968 WARN [M:0;jenkins-hbase5:37087] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 15:40:50,968 INFO [M:0;jenkins-hbase5:37087] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 15:40:50,968 INFO [M:0;jenkins-hbase5:37087] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 15:40:50,969 DEBUG [M:0;jenkins-hbase5:37087] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 15:40:50,969 INFO [M:0;jenkins-hbase5:37087] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:40:50,969 DEBUG [M:0;jenkins-hbase5:37087] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:40:50,969 DEBUG [M:0;jenkins-hbase5:37087] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 15:40:50,969 DEBUG [M:0;jenkins-hbase5:37087] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:40:50,969 INFO [M:0;jenkins-hbase5:37087] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=185.36 KB heapSize=221.74 KB 2023-07-12 15:40:51,378 INFO [M:0;jenkins-hbase5:37087] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=185.36 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fe75d809f1f44516821e11df6a8fa48d 2023-07-12 15:40:51,382 INFO [M:0;jenkins-hbase5:37087] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fe75d809f1f44516821e11df6a8fa48d 2023-07-12 15:40:51,383 DEBUG [M:0;jenkins-hbase5:37087] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fe75d809f1f44516821e11df6a8fa48d as hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fe75d809f1f44516821e11df6a8fa48d 2023-07-12 15:40:51,387 INFO [M:0;jenkins-hbase5:37087] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fe75d809f1f44516821e11df6a8fa48d 2023-07-12 15:40:51,387 INFO [M:0;jenkins-hbase5:37087] regionserver.HStore(1080): Added hdfs://localhost:38173/user/jenkins/test-data/a719441c-55e7-157d-9f80-1a2eab930788/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fe75d809f1f44516821e11df6a8fa48d, entries=53, sequenceid=421, filesize=13.6 K 2023-07-12 15:40:51,388 INFO [M:0;jenkins-hbase5:37087] regionserver.HRegion(2948): Finished flush of dataSize ~185.36 KB/189804, heapSize ~221.73 KB/227048, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=421, compaction requested=false 2023-07-12 15:40:51,389 INFO [M:0;jenkins-hbase5:37087] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 15:40:51,389 DEBUG [M:0;jenkins-hbase5:37087] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 15:40:51,391 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 15:40:51,391 INFO [M:0;jenkins-hbase5:37087] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 15:40:51,392 INFO [M:0;jenkins-hbase5:37087] ipc.NettyRpcServer(158): Stopping server on /172.31.10.131:37087 2023-07-12 15:40:51,393 DEBUG [M:0;jenkins-hbase5:37087] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase5.apache.org,37087,1689176380200 already deleted, retry=false 2023-07-12 15:40:51,761 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:40:51,761 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): master:37087-0x101c5ccda360000, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:40:51,761 INFO [M:0;jenkins-hbase5:37087] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,37087,1689176380200; zookeeper connection closed. 2023-07-12 15:40:51,861 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:40:51,861 DEBUG [Listener at localhost/38241-EventThread] zookeeper.ZKWatcher(600): regionserver:45089-0x101c5ccda360001, quorum=127.0.0.1:51623, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 15:40:51,861 INFO [RS:0;jenkins-hbase5:45089] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase5.apache.org,45089,1689176381369; zookeeper connection closed. 2023-07-12 15:40:51,862 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@174e7840] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@174e7840 2023-07-12 15:40:51,862 INFO [Listener at localhost/38241] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-07-12 15:40:51,862 WARN [Listener at localhost/38241] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 15:40:51,865 INFO [Listener at localhost/38241] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:40:51,970 WARN [BP-1556192786-172.31.10.131-1689176377226 heartbeating to localhost/127.0.0.1:38173] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 15:40:51,970 WARN [BP-1556192786-172.31.10.131-1689176377226 heartbeating to localhost/127.0.0.1:38173] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1556192786-172.31.10.131-1689176377226 (Datanode Uuid a2d3ebee-8da6-4387-bf6f-5ce1660e5ee1) service to localhost/127.0.0.1:38173 2023-07-12 15:40:51,972 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/cluster_25c97bd4-4232-2ad9-6328-0c863aed8e02/dfs/data/data1/current/BP-1556192786-172.31.10.131-1689176377226] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:40:51,972 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/86226c37-b328-6b87-1526-f6e07c7428ff/cluster_25c97bd4-4232-2ad9-6328-0c863aed8e02/dfs/data/data2/current/BP-1556192786-172.31.10.131-1689176377226] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 15:40:52,008 INFO [Listener at localhost/38241] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 15:40:52,122 INFO [Listener at localhost/38241] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 15:40:52,169 INFO [Listener at localhost/38241] hbase.HBaseTestingUtility(1293): Minicluster is down